2023-12-02T22:03:41.838 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-02T22:03:41.838 DEBUG:teuthology.run:Teuthology command: teuthology --name dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi --owner scheduled_dis@teuthology --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} --archive /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805 -- /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/orig.config.yaml 2023-12-02T22:03:41.890 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-02T22:03:41.985 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: idryomov@gmail.com first_in_suite: false job_id: '7475805' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 owner: scheduled_dis@teuthology priority: 69 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2667 sha1: 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 sleep_before_teardown: 0 subset: 17/99 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXvhR6JCsTwPdNXSIOZvx45StfGsMUDwgQgZDSRvgDFGOjSjKQY32w4gnwhnr3iQ8P2nxx3y/yZOma8AInV3tM= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAO9CBiefD2/lvxCVlSKINHzBVGThnPqas2AV49ObW4POGhsYMBoxblDlHCPGauDDzzxu0VRdL2bZtXX0ZVQWQ0= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-02_21:54:05 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-02T22:03:41.985 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa; will attempt to use it 2023-12-02T22:03:41.986 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks 2023-12-02T22:03:41.986 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-02T22:03:41.988 INFO:teuthology.task.internal:Checking packages... 2023-12-02T22:03:42.009 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95' 2023-12-02T22:03:42.009 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-02T22:03:42.009 INFO:teuthology.packaging:ref: None 2023-12-02T22:03:42.009 INFO:teuthology.packaging:tag: None 2023-12-02T22:03:42.009 INFO:teuthology.packaging:branch: wip-dis-testing 2023-12-02T22:03:42.009 INFO:teuthology.packaging:sha1: 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:03:42.010 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing 2023-12-02T22:03:42.158 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7670.g0c523c70 2023-12-02T22:03:42.158 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-02T22:03:42.165 INFO:teuthology.task.internal:no buildpackages task found 2023-12-02T22:03:42.165 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-02T22:03:42.175 INFO:teuthology.task.internal:Saving configuration 2023-12-02T22:03:42.195 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-02T22:03:42.201 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T22:03:42.226 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 21:55:17.244701', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXvhR6JCsTwPdNXSIOZvx45StfGsMUDwgQgZDSRvgDFGOjSjKQY32w4gnwhnr3iQ8P2nxx3y/yZOma8AInV3tM='} 2023-12-02T22:03:42.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 21:55:17.246069', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAO9CBiefD2/lvxCVlSKINHzBVGThnPqas2AV49ObW4POGhsYMBoxblDlHCPGauDDzzxu0VRdL2bZtXX0ZVQWQ0='} 2023-12-02T22:03:42.250 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-02T22:03:42.256 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-02T22:03:42.256 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-02T22:03:42.256 INFO:teuthology.run_tasks:Running task console_log... 2023-12-02T22:03:42.326 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f34326ad9d0>, signals=[15]) 2023-12-02T22:03:42.326 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-02T22:03:42.333 INFO:teuthology.task.internal:Opening connections... 2023-12-02T22:03:42.333 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T22:03:42.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:03:42.433 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:03:42.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:03:42.520 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-02T22:03:42.526 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2023-12-02T22:03:42.544 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2023-12-02T22:03:42.544 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2023-12-02T22:03:42.601 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-02T22:03:42.602 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-02T22:03:42.602 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-02T22:03:42.602 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2023-12-02T22:03:42.625 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-12-02T22:03:42.643 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-12-02T22:03:42.643 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-12-02T22:03:42.702 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-02T22:03:42.703 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-02T22:03:42.704 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-12-02T22:03:42.724 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-02T22:03:42.732 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-02T22:03:42.737 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-02T22:03:42.737 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2023-12-02T22:03:42.740 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-12-02T22:03:42.762 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-02T22:03:42.767 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-02T22:03:42.767 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2023-12-02T22:03:42.797 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-12-02T22:03:42.829 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-02T22:03:42.887 INFO:teuthology.run_tasks:Running task kernel... 2023-12-02T22:03:42.901 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-02T22:03:42.901 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-02T22:03:42.901 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-02T22:03:42.902 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-02T22:03:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T22:03:42.917 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-12-02T22:03:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T22:03:42.918 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-12-02T22:03:42.975 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2023-12-02T22:03:42.976 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-361.el9.x86_64 2023-12-02T22:03:42.976 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2023-12-02T22:03:42.977 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2023-12-02T22:03:42.977 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2023-12-02T22:03:42.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2023-12-02T22:03:44.999 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-02T22:03:45.132 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-02T22:03:46.365 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2023-12-02T22:03:46.557 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2023-12-02T22:03:47.848 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 7.9 MB 00:00 2023-12-02T22:03:48.139 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-12-02T22:03:49.941 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 19 MB/s | 18 MB 00:00 2023-12-02T22:03:51.466 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 18 MB 00:02 2023-12-02T22:03:53.999 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-12-02T22:03:55.417 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 15 kB 00:00 2023-12-02T22:03:55.925 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 5.8 MB 00:01 2023-12-02T22:03:57.608 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2023-12-02T22:03:59.087 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-12-02T22:04:02.167 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 20 MB 00:06 2023-12-02T22:04:03.868 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-02T22:04:06.216 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-02T22:04:06.216 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-02T22:04:06.305 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-02T22:04:06.313 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:04:06.314 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 96 M 2023-12-02T22:04:06.315 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 121 M 2023-12-02T22:04:06.315 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:04:07.149 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2023-12-02T22:04:07.644 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 4.5 MB/s | 5.6 MB 00:01 2023-12-02T22:04:09.111 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 7.5 MB/s | 20 MB 00:02 2023-12-02T22:04:09.623 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-02T22:04:09.623 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-02T22:04:09.707 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2023-12-02T22:04:09.717 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:04:09.718 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 96 M 2023-12-02T22:04:09.718 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 121 M 2023-12-02T22:04:09.718 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:04:15.008 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 1.1 MB/s | 5.6 MB 00:05 2023-12-02T22:04:16.744 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 3.7 MB/s | 38 MB 00:10 2023-12-02T22:04:17.902 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 3.2 MB/s | 32 MB 00:10 2023-12-02T22:04:17.902 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:04:17.902 INFO:teuthology.orchestra.run.smithi170.stdout:Total 8.3 MB/s | 96 MB 00:11 2023-12-02T22:04:18.767 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:04:19.127 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:04:19.127 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:04:20.462 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:04:20.462 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:04:20.691 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 1.9 MB/s | 20 MB 00:10 2023-12-02T22:04:22.899 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-core-5.14.0-388.el9.x86_6 4.1 MB/s | 32 MB 00:07 2023-12-02T22:04:23.241 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 2.8 MB/s | 38 MB 00:13 2023-12-02T22:04:23.241 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:04:23.241 INFO:teuthology.orchestra.run.smithi012.stdout:Total 7.1 MB/s | 96 MB 00:13 2023-12-02T22:04:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:04:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-02T22:04:24.131 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:04:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T22:04:24.554 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:04:24.554 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:04:25.930 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:04:25.930 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:04:26.939 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T22:04:27.115 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T22:04:28.722 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:04:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-02T22:04:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T22:04:31.774 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T22:04:31.981 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:04:32.433 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T22:04:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T22:04:36.631 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:04:37.555 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T22:04:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:04:42.660 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:05.699 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:05.742 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:06.193 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:06.193 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-02T22:05:06.193 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T22:05:06.193 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T22:05:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:06.572 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:05:06.572 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:05:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-02T22:05:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-02T22:05:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-02T22:05:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-02T22:05:06.573 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:05:06.573 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:05:07.050 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T22:05:07.202 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T22:05:07.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T22:05:07.202 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-02T22:05:07.202 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-02T22:05:07.202 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-02T22:05:07.202 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-02T22:05:07.203 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2023-12-02T22:05:07.203 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T22:05:07.319 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T22:05:07.320 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T22:05:07.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2023-12-02T22:05:07.360 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-02T22:05:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-02T22:05:07.362 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-02T22:05:07.363 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2023-12-02T22:05:07.363 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:05:07.363 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2023-12-02T22:05:07.364 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-02T22:05:07.364 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-388.el9.x86_64 2023-12-02T22:05:07.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-02T22:05:08.040 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-12-02T22:05:09.382 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-02T22:05:09.434 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-12-02T22:05:09.437 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-12-02T22:05:09.452 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.TJnaGUsr82 2023-12-02T22:05:09.452 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TJnaGUsr82 2023-12-02T22:05:09.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.TJnaGUsr82 2023-12-02T22:05:09.665 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.TJnaGUsr82 is 6KB 2023-12-02T22:05:09.685 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.TJnaGUsr82 2023-12-02T22:05:09.699 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2023-12-02T22:05:09.766 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-02T22:05:09.766 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-02T22:05:09.767 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-02T22:05:09.767 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-02T22:05:09.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-02T22:05:09.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-12-02T22:05:13.110 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:13.154 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:13.616 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:13.617 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-02T22:05:13.617 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T22:05:13.617 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T22:05:14.093 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T22:05:14.093 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:05:14.094 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:05:14.579 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T22:05:14.730 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T22:05:14.731 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T22:05:14.731 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-02T22:05:14.731 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-02T22:05:14.731 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-02T22:05:14.731 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-02T22:05:14.731 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2023-12-02T22:05:14.731 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T22:05:14.855 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T22:05:14.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T22:05:14.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-02T22:05:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:05:14.903 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2023-12-02T22:05:14.904 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-02T22:05:14.904 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-388.el9.x86_64 2023-12-02T22:05:14.905 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-02T22:05:15.545 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2023-12-02T22:05:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-02T22:05:16.961 INFO:teuthology.orchestra.run.smithi012.stderr:done 2023-12-02T22:05:16.963 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-12-02T22:05:16.978 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.rrUcWeg8pL 2023-12-02T22:05:16.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rrUcWeg8pL 2023-12-02T22:05:17.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.rrUcWeg8pL 2023-12-02T22:05:17.200 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.rrUcWeg8pL is 6KB 2023-12-02T22:05:17.220 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.rrUcWeg8pL 2023-12-02T22:05:17.235 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2023-12-02T22:05:17.314 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-02T22:05:17.314 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-02T22:05:17.314 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-02T22:05:17.314 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-02T22:05:17.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-02T22:05:17.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2023-12-02T22:05:39.946 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-02T22:05:39.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-02T22:05:39.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:05:47.482 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-02T22:05:47.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-02T22:05:47.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:06:22.051 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-12-02T22:06:31.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-02T22:06:31.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:06:34.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-12-02T22:06:39.955 DEBUG:teuthology.orchestra.remote:timed out 2023-12-02T22:06:46.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-02T22:06:46.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:06:46.495 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-12-02T22:06:47.014 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-02T22:06:47.014 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-02T22:06:47.014 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-12-02T22:06:47.029 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-388.el9.x86_64 2023-12-02T22:06:47.029 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-02T22:06:47.029 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-02T22:06:47.029 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-02T22:06:48.030 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-02T22:06:48.030 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-02T22:06:48.141 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2023-12-02T22:06:48.185 DEBUG:teuthology.parallel:result is None 2023-12-02T22:06:48.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-02T22:06:48.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:06:50.392 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-12-02T22:06:50.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-02T22:06:50.907 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-02T22:06:50.907 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-12-02T22:06:50.922 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-388.el9.x86_64 2023-12-02T22:06:50.922 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-02T22:06:50.922 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-02T22:06:50.922 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-02T22:06:51.923 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-02T22:06:51.923 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-02T22:06:52.016 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2023-12-02T22:06:52.064 DEBUG:teuthology.parallel:result is None 2023-12-02T22:06:52.064 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-02T22:06:52.071 INFO:teuthology.task.internal:Creating test directory... 2023-12-02T22:06:52.071 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-02T22:06:52.074 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-02T22:06:52.089 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-02T22:06:52.095 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-02T22:06:52.105 INFO:teuthology.task.internal:Creating archive directory... 2023-12-02T22:06:52.105 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-02T22:06:52.128 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-02T22:06:52.152 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-02T22:06:52.159 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-02T22:06:52.159 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-02T22:06:52.181 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-02T22:06:52.219 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T22:06:52.221 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T22:06:52.234 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T22:06:52.234 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T22:06:52.236 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-02T22:06:52.242 INFO:teuthology.task.internal:Configuring sudo... 2023-12-02T22:06:52.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-02T22:06:52.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-02T22:06:52.303 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-02T22:06:52.312 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-02T22:06:52.313 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-02T22:06:52.343 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-02T22:06:52.359 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T22:06:52.436 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T22:06:52.494 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T22:06:52.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T22:06:52.579 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:06:52.579 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-02T22:06:52.648 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T22:06:52.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T22:06:52.737 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T22:06:52.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T22:06:52.861 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:06:52.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-02T22:06:52.930 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2023-12-02T22:06:52.932 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-12-02T22:06:52.990 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T22:06:53.022 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T22:06:53.402 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-02T22:06:53.411 INFO:teuthology.task.internal:Starting timer... 2023-12-02T22:06:53.411 INFO:teuthology.run_tasks:Running task pcp... 2023-12-02T22:06:53.423 INFO:teuthology.run_tasks:Running task selinux... 2023-12-02T22:06:53.432 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2023-12-02T22:06:53.498 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2023-12-02T22:06:53.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2023-12-02T22:06:53.551 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2023-12-02T22:06:53.553 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-02T22:06:53.553 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2023-12-02T22:06:53.577 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2023-12-02T22:06:53.578 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2023-12-02T22:06:53.610 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2023-12-02T22:06:53.611 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2023-12-02T22:06:53.611 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T22:06:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T22:06:53.648 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T22:06:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T22:06:53.678 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-02T22:06:53.678 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2023-12-02T22:06:53.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2023-12-02T22:06:53.754 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-02T22:06:53.764 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-02T22:06:53.765 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-02T22:06:53.790 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-02T22:06:53.792 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-02T22:18:07.033 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2023-12-02T22:18:07.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-02T22:18:07.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:18:07.133 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-12-02T22:18:07.202 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-02T22:18:07.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-02T22:18:07.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T22:18:07.286 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-12-02T22:18:07.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-02T22:18:07.353 INFO:teuthology.run_tasks:Running task clock... 2023-12-02T22:18:07.365 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-02T22:18:07.365 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-02T22:18:07.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T22:18:07.368 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-02T22:18:07.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T22:18:07.410 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-02T22:18:07.425 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-02T22:18:07.431 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-02T22:18:07.442 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-02T22:18:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2023-12-02T22:18:07.471 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2023-12-02T22:18:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2023-12-02T22:18:07.484 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2023-12-02T22:18:07.501 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-02T22:18:07.507 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-02T22:18:07.517 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-02T22:18:07.524 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-02T22:18:07.560 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2023-12-02T22:18:07.566 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T22:18:07.566 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-12-02T22:18:07.566 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.566 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.567 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.567 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.572 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2023-12-02T22:18:07.579 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T22:18:07.579 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2023-12-02T22:18:07.580 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.580 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.580 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.580 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T22:18:07.580 INFO:teuthology.run_tasks:Running task install... 2023-12-02T22:18:07.596 DEBUG:teuthology.task.install:project ceph 2023-12-02T22:18:07.596 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}} 2023-12-02T22:18:07.596 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'} 2023-12-02T22:18:07.596 INFO:teuthology.task.install:Using flavor: default 2023-12-02T22:18:07.610 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-02T22:18:07.610 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-02T22:18:07.611 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95', 'tag': None, 'wait_for_package': False} 2023-12-02T22:18:07.611 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:18:07.613 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95', 'tag': None, 'wait_for_package': False} 2023-12-02T22:18:07.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:18:07.766 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/ 2023-12-02T22:18:07.766 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7670.g0c523c70 2023-12-02T22:18:07.793 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/ 2023-12-02T22:18:07.793 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7670.g0c523c70 2023-12-02T22:18:07.913 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-02T22:18:07.914 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:18:07.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-02T22:18:07.941 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-02T22:18:07.941 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:18:07.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-02T22:18:07.950 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-02T22:18:07.950 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-02T22:18:07.971 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-02T22:18:07.971 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-02T22:18:08.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-02T22:18:08.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-02T22:18:08.114 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-02T22:18:08.118 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-02T22:18:08.144 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2023-12-02T22:18:08.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2023-12-02T22:18:08.184 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2023-12-02T22:18:08.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2023-12-02T22:18:08.401 INFO:teuthology.orchestra.run.smithi012.stdout:73 files removed 2023-12-02T22:18:08.422 INFO:teuthology.orchestra.run.smithi170.stdout:73 files removed 2023-12-02T22:18:08.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw 2023-12-02T22:18:08.449 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2023-12-02T22:18:09.119 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 222 kB/s | 78 kB 00:00 2023-12-02T22:18:09.245 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 164 kB/s | 78 kB 00:00 2023-12-02T22:18:09.444 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 34 kB/s | 9.5 kB 00:00 2023-12-02T22:18:09.612 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 30 kB/s | 9.5 kB 00:00 2023-12-02T22:18:09.763 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2023-12-02T22:18:09.963 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2023-12-02T22:18:10.415 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-02T22:18:10.739 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.4 MB 00:00 2023-12-02T22:18:12.064 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-12-02T22:18:12.149 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2023-12-02T22:18:13.240 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 7.9 MB 00:00 2023-12-02T22:18:14.054 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 7.9 MB 00:01 2023-12-02T22:18:15.316 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 20 MB/s | 18 MB 00:00 2023-12-02T22:18:17.015 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 18 MB 00:01 2023-12-02T22:18:20.224 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 5.8 MB 00:01 2023-12-02T22:18:21.656 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 5.8 MB 00:01 2023-12-02T22:18:21.875 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-02T22:18:23.274 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 15 kB 00:00 2023-12-02T22:18:24.694 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-12-02T22:18:25.963 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 20 MB 00:03 2023-12-02T22:18:29.392 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-02T22:18:30.917 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-02T22:18:32.058 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:18:32.062 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 8.6 M 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 3.3 M 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 3.0 M 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 5.2 M 2023-12-02T22:18:32.063 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 19 M 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 24 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 718 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 476 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 4.8 M 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-02T22:18:32.064 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 45 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 129 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 161 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 321 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 297 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 99 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout:Install 27 Packages 2023-12-02T22:18:32.065 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2023-12-02T22:18:32.066 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:32.066 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 55 M 2023-12-02T22:18:32.066 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:18:32.743 INFO:teuthology.orchestra.run.smithi170.stdout:(1/29): ceph-base-18.0.0-7670.g0c523c70.el9.x86 8.8 MB/s | 5.2 MB 00:00 2023-12-02T22:18:32.802 INFO:teuthology.orchestra.run.smithi170.stdout:(2/29): ceph-selinux-18.0.0-7670.g0c523c70.el9. 413 kB/s | 24 kB 00:00 2023-12-02T22:18:32.902 INFO:teuthology.orchestra.run.smithi170.stdout:(3/29): libcephfs2-18.0.0-7670.g0c523c70.el9.x8 7.0 MB/s | 718 kB 00:00 2023-12-02T22:18:33.002 INFO:teuthology.orchestra.run.smithi170.stdout:(4/29): libradosstriper1-18.0.0-7670.g0c523c70. 4.7 MB/s | 476 kB 00:00 2023-12-02T22:18:33.436 INFO:teuthology.orchestra.run.smithi170.stdout:(5/29): librgw2-18.0.0-7670.g0c523c70.el9.x86_6 11 MB/s | 4.8 MB 00:00 2023-12-02T22:18:33.486 INFO:teuthology.orchestra.run.smithi170.stdout:(6/29): python3-ceph-argparse-18.0.0-7670.g0c52 908 kB/s | 45 kB 00:00 2023-12-02T22:18:33.552 INFO:teuthology.orchestra.run.smithi170.stdout:(7/29): python3-ceph-common-18.0.0-7670.g0c523c 1.9 MB/s | 129 kB 00:00 2023-12-02T22:18:33.616 INFO:teuthology.orchestra.run.smithi170.stdout:(8/29): python3-cephfs-18.0.0-7670.g0c523c70.el 2.5 MB/s | 161 kB 00:00 2023-12-02T22:18:33.765 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:18:33.781 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:18:33.781 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:18:33.781 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:18:33.781 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:18:33.781 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 8.6 M 2023-12-02T22:18:33.781 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 3.3 M 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 3.0 M 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 5.2 M 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 19 M 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 24 k 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 718 k 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-02T22:18:33.782 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 476 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 4.8 M 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 45 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 129 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 161 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 321 k 2023-12-02T22:18:33.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 297 k 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 99 k 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout:Install 27 Packages 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 2 Packages 2023-12-02T22:18:33.784 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:33.785 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 55 M 2023-12-02T22:18:33.785 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:18:33.866 INFO:teuthology.orchestra.run.smithi170.stdout:(9/29): ceph-common-18.0.0-7670.g0c523c70.el9.x 11 MB/s | 19 MB 00:01 2023-12-02T22:18:33.891 INFO:teuthology.orchestra.run.smithi170.stdout:(10/29): python3-rados-18.0.0-7670.g0c523c70.el 1.1 MB/s | 321 kB 00:00 2023-12-02T22:18:33.917 INFO:teuthology.orchestra.run.smithi170.stdout:(11/29): python3-rbd-18.0.0-7670.g0c523c70.el9. 5.8 MB/s | 297 kB 00:00 2023-12-02T22:18:33.934 INFO:teuthology.orchestra.run.smithi170.stdout:(12/29): python3-rgw-18.0.0-7670.g0c523c70.el9. 2.3 MB/s | 99 kB 00:00 2023-12-02T22:18:34.059 INFO:teuthology.orchestra.run.smithi170.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2023-12-02T22:18:34.085 INFO:teuthology.orchestra.run.smithi170.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-12-02T22:18:34.110 INFO:teuthology.orchestra.run.smithi170.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2023-12-02T22:18:34.227 INFO:teuthology.orchestra.run.smithi170.stdout:(16/29): ceph-radosgw-18.0.0-7670.g0c523c70.el9 4.1 MB/s | 8.6 MB 00:02 2023-12-02T22:18:34.261 INFO:teuthology.orchestra.run.smithi170.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2023-12-02T22:18:34.286 INFO:teuthology.orchestra.run.smithi170.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 282 kB 00:00 2023-12-02T22:18:34.303 INFO:teuthology.orchestra.run.smithi170.stdout:(19/29): python3-prettytable-0.7.2-27.el9.noarc 551 kB/s | 41 kB 00:00 2023-12-02T22:18:34.428 INFO:teuthology.orchestra.run.smithi170.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 195 kB/s | 24 kB 00:00 2023-12-02T22:18:34.537 INFO:teuthology.orchestra.run.smithi170.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2023-12-02T22:18:34.570 INFO:teuthology.orchestra.run.smithi170.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2023-12-02T22:18:34.629 INFO:teuthology.orchestra.run.smithi170.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 734 kB/s | 67 kB 00:00 2023-12-02T22:18:34.712 INFO:teuthology.orchestra.run.smithi170.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 5.8 MB/s | 837 kB 00:00 2023-12-02T22:18:34.738 INFO:teuthology.orchestra.run.smithi170.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-12-02T22:18:34.805 INFO:teuthology.orchestra.run.smithi170.stdout:(26/29): libarrow-9.0.0-9.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2023-12-02T22:18:34.856 INFO:teuthology.orchestra.run.smithi170.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-12-02T22:18:35.132 INFO:teuthology.orchestra.run.smithi170.stdout:(28/29): librbd1-18.0.0-7670.g0c523c70.el9.x86_ 9.3 MB/s | 3.0 MB 00:00 2023-12-02T22:18:35.160 INFO:teuthology.orchestra.run.smithi012.stdout:(1/29): ceph-common-18.0.0-7670.g0c523c70.el9.x 16 MB/s | 19 MB 00:01 2023-12-02T22:18:35.210 INFO:teuthology.orchestra.run.smithi012.stdout:(2/29): ceph-selinux-18.0.0-7670.g0c523c70.el9. 484 kB/s | 24 kB 00:00 2023-12-02T22:18:35.285 INFO:teuthology.orchestra.run.smithi012.stdout:(3/29): libcephfs2-18.0.0-7670.g0c523c70.el9.x8 9.4 MB/s | 718 kB 00:00 2023-12-02T22:18:35.352 INFO:teuthology.orchestra.run.smithi012.stdout:(4/29): libradosstriper1-18.0.0-7670.g0c523c70. 7.0 MB/s | 476 kB 00:00 2023-12-02T22:18:35.522 INFO:teuthology.orchestra.run.smithi170.stdout:(29/29): librados2-18.0.0-7670.g0c523c70.el9.x8 4.2 MB/s | 3.3 MB 00:00 2023-12-02T22:18:35.526 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:18:35.527 INFO:teuthology.orchestra.run.smithi170.stdout:Total 16 MB/s | 55 MB 00:03 2023-12-02T22:18:35.527 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:18:35.562 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:18:35.562 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:18:35.678 INFO:teuthology.orchestra.run.smithi012.stdout:(5/29): librgw2-18.0.0-7670.g0c523c70.el9.x86_6 15 MB/s | 4.8 MB 00:00 2023-12-02T22:18:35.719 INFO:teuthology.orchestra.run.smithi012.stdout:(6/29): python3-ceph-argparse-18.0.0-7670.g0c52 1.1 MB/s | 45 kB 00:00 2023-12-02T22:18:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:(7/29): python3-ceph-common-18.0.0-7670.g0c523c 3.0 MB/s | 129 kB 00:00 2023-12-02T22:18:35.828 INFO:teuthology.orchestra.run.smithi012.stdout:(8/29): ceph-base-18.0.0-7670.g0c523c70.el9.x86 2.8 MB/s | 5.2 MB 00:01 2023-12-02T22:18:35.854 INFO:teuthology.orchestra.run.smithi012.stdout:(9/29): python3-cephfs-18.0.0-7670.g0c523c70.el 1.7 MB/s | 161 kB 00:00 2023-12-02T22:18:35.896 INFO:teuthology.orchestra.run.smithi012.stdout:(10/29): python3-rados-18.0.0-7670.g0c523c70.el 4.7 MB/s | 321 kB 00:00 2023-12-02T22:18:35.921 INFO:teuthology.orchestra.run.smithi012.stdout:(11/29): python3-rbd-18.0.0-7670.g0c523c70.el9. 4.4 MB/s | 297 kB 00:00 2023-12-02T22:18:35.980 INFO:teuthology.orchestra.run.smithi012.stdout:(12/29): python3-rgw-18.0.0-7670.g0c523c70.el9. 1.2 MB/s | 99 kB 00:00 2023-12-02T22:18:36.014 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:18:36.014 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:18:36.030 INFO:teuthology.orchestra.run.smithi012.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 975 kB/s | 106 kB 00:00 2023-12-02T22:18:36.057 INFO:teuthology.orchestra.run.smithi012.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2023-12-02T22:18:36.080 INFO:teuthology.orchestra.run.smithi012.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2023-12-02T22:18:36.114 INFO:teuthology.orchestra.run.smithi012.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-02T22:18:36.141 INFO:teuthology.orchestra.run.smithi012.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.6 MB/s | 282 kB 00:00 2023-12-02T22:18:36.167 INFO:teuthology.orchestra.run.smithi012.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 793 kB/s | 41 kB 00:00 2023-12-02T22:18:36.802 INFO:teuthology.orchestra.run.smithi012.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 469 kB/s | 309 kB 00:00 2023-12-02T22:18:36.901 INFO:teuthology.orchestra.run.smithi012.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 244 kB/s | 24 kB 00:00 2023-12-02T22:18:37.004 INFO:teuthology.orchestra.run.smithi012.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 482 kB/s | 49 kB 00:00 2023-12-02T22:18:37.102 INFO:teuthology.orchestra.run.smithi012.stdout:(22/29): ceph-radosgw-18.0.0-7670.g0c523c70.el9 2.7 MB/s | 8.6 MB 00:03 2023-12-02T22:18:37.127 INFO:teuthology.orchestra.run.smithi012.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2023-12-02T22:18:37.219 INFO:teuthology.orchestra.run.smithi012.stdout:(24/29): libarrow-9.0.0-9.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2023-12-02T22:18:37.269 INFO:teuthology.orchestra.run.smithi012.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2023-12-02T22:18:37.368 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:18:37.378 INFO:teuthology.orchestra.run.smithi012.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2023-12-02T22:18:37.461 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-02T22:18:37.493 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-02T22:18:37.552 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-02T22:18:37.663 INFO:teuthology.orchestra.run.smithi012.stdout:(27/29): librbd1-18.0.0-7670.g0c523c70.el9.x86_ 11 MB/s | 3.0 MB 00:00 2023-12-02T22:18:37.865 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-02T22:18:37.912 INFO:teuthology.orchestra.run.smithi012.stdout:(28/29): librados2-18.0.0-7670.g0c523c70.el9.x8 5.2 MB/s | 3.3 MB 00:00 2023-12-02T22:18:37.917 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 5/31 2023-12-02T22:18:37.945 INFO:teuthology.orchestra.run.smithi012.stdout:(29/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 993 kB/s | 837 kB 00:00 2023-12-02T22:18:37.952 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:18:37.952 INFO:teuthology.orchestra.run.smithi012.stdout:Total 13 MB/s | 55 MB 00:04 2023-12-02T22:18:37.952 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:18:37.995 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:18:37.995 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:18:38.000 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 5/31 2023-12-02T22:18:38.094 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 6/31 2023-12-02T22:18:38.119 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 7/31 2023-12-02T22:18:38.181 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 7/31 2023-12-02T22:18:38.214 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-02T22:18:38.245 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:18.0.0-7670.g0c523c70.el9. 9/31 2023-12-02T22:18:38.314 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 10/31 2023-12-02T22:18:38.338 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_6 11/31 2023-12-02T22:18:38.403 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_6 11/31 2023-12-02T22:18:38.477 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:18:38.477 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:18:38.509 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-02T22:18:38.554 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-02T22:18:38.595 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-02T22:18:38.628 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-02T22:18:38.656 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-02T22:18:38.689 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-02T22:18:38.992 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-02T22:18:39.020 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 19/31 2023-12-02T22:18:39.100 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 19/31 2023-12-02T22:18:39.132 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 20/31 2023-12-02T22:18:39.168 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-02T22:18:39.594 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-02T22:18:39.622 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 23/31 2023-12-02T22:18:39.685 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 23/31 2023-12-02T22:18:39.761 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 24/31 2023-12-02T22:18:39.791 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:18.0.0-7670.g0c523c70.el9.x8 25/31 2023-12-02T22:18:39.856 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:18:40.047 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-02T22:18:40.084 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-02T22:18:40.152 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-02T22:18:40.493 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-02T22:18:40.567 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 5/31 2023-12-02T22:18:40.667 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 5/31 2023-12-02T22:18:40.754 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 6/31 2023-12-02T22:18:40.903 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 7/31 2023-12-02T22:18:40.949 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 7/31 2023-12-02T22:18:40.976 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-02T22:18:41.012 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:18.0.0-7670.g0c523c70.el9. 9/31 2023-12-02T22:18:41.079 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 10/31 2023-12-02T22:18:41.103 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_6 11/31 2023-12-02T22:18:41.169 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_6 11/31 2023-12-02T22:18:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-02T22:18:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-02T22:18:41.352 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-02T22:18:41.383 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-02T22:18:41.412 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-02T22:18:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-02T22:18:41.777 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-02T22:18:41.812 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 19/31 2023-12-02T22:18:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 26/31 2023-12-02T22:18:41.903 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 19/31 2023-12-02T22:18:41.913 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 26/31 2023-12-02T22:18:41.933 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 20/31 2023-12-02T22:18:41.969 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-02T22:18:42.418 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-02T22:18:42.448 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 23/31 2023-12-02T22:18:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 26/31 2023-12-02T22:18:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 27/31 2023-12-02T22:18:42.522 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 23/31 2023-12-02T22:18:42.572 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 27/31 2023-12-02T22:18:42.572 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-02T22:18:42.573 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-02T22:18:42.573 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:42.597 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 24/31 2023-12-02T22:18:42.609 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 28/31 2023-12-02T22:18:42.783 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:18.0.0-7670.g0c523c70.el9.x8 25/31 2023-12-02T22:18:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 26/31 2023-12-02T22:18:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 26/31 2023-12-02T22:18:45.591 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 26/31 2023-12-02T22:18:45.635 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 27/31 2023-12-02T22:18:45.704 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 27/31 2023-12-02T22:18:45.704 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-02T22:18:45.704 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-02T22:18:45.704 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:45.753 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 28/31 2023-12-02T22:18:52.210 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 28/31 2023-12-02T22:18:52.210 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2023-12-02T22:18:52.210 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2023-12-02T22:18:52.210 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2023-12-02T22:18:52.210 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2023-12-02T22:18:52.210 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2023-12-02T22:18:52.211 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2023-12-02T22:18:52.211 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2023-12-02T22:18:52.211 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:53.009 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 29/31 2023-12-02T22:18:53.034 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 29/31 2023-12-02T22:18:53.034 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:18:53.034 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-02T22:18:53.034 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T22:18:53.034 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T22:18:53.035 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:53.036 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T22:18:53.128 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T22:18:53.129 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T22:18:54.215 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T22:18:54.215 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 1/31 2023-12-02T22:18:54.215 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 2/31 2023-12-02T22:18:54.215 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 3/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 4/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 5/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_6 6/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 7/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7670.g0c523c70.el9. 8/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:18.0.0-7670.g0c523c70.el9.x8 9/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 10/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 11/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 12/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 13/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-02T22:18:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 28/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-02T22:18:54.217 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 30/31 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.392 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:55.394 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:18:55.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2023-12-02T22:18:55.787 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 28/31 2023-12-02T22:18:55.787 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2023-12-02T22:18:55.787 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2023-12-02T22:18:55.788 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2023-12-02T22:18:55.788 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2023-12-02T22:18:55.788 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2023-12-02T22:18:55.788 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2023-12-02T22:18:55.788 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2023-12-02T22:18:55.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:56.309 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:27 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:18:56.453 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:18:56.453 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:18:56.453 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:18:56.453 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:18:56.453 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:18:56.453 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 41 M 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 163 k 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-02T22:18:56.454 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:18:56.455 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 42 M 2023-12-02T22:18:56.455 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 173 M 2023-12-02T22:18:56.455 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:18:56.618 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-02T22:18:56.625 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 29/31 2023-12-02T22:18:56.651 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 29/31 2023-12-02T22:18:56.651 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:18:56.651 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-02T22:18:56.651 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T22:18:56.651 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T22:18:56.651 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:56.653 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T22:18:56.660 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-02T22:18:56.702 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-02T22:18:56.737 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-02T22:18:56.752 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T22:18:56.752 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T22:18:56.827 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): libcephsqlite-18.0.0-7670.g0c523c70.el9. 440 kB/s | 163 kB 00:00 2023-12-02T22:18:57.875 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 1/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 2/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 3/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 4/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 5/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_6 6/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 7/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7670.g0c523c70.el9. 8/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:18.0.0-7670.g0c523c70.el9.x8 9/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 10/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 11/31 2023-12-02T22:18:57.876 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 12/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 13/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-02T22:18:57.877 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-02T22:18:57.878 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 28/31 2023-12-02T22:18:57.878 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-02T22:18:57.878 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 30/31 2023-12-02T22:18:58.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-02T22:18:58.945 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:58.947 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:18:59.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-test 2023-12-02T22:18:59.594 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-18.0.0-7670.g0c523c70.el9.x86_ 13 MB/s | 41 MB 00:03 2023-12-02T22:18:59.597 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:18:59.597 INFO:teuthology.orchestra.run.smithi170.stdout:Total 13 MB/s | 42 MB 00:03 2023-12-02T22:18:59.598 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:18:59.617 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:18:59.796 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:29 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:18:59.947 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 41 M 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 163 k 2023-12-02T22:18:59.948 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2023-12-02T22:18:59.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:18:59.950 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 42 M 2023-12-02T22:18:59.950 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 173 M 2023-12-02T22:18:59.950 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:18:59.977 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:18:59.977 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:00.160 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 902 kB/s | 186 kB 00:00 2023-12-02T22:19:00.216 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): libcephsqlite-18.0.0-7670.g0c523c70.el9. 617 kB/s | 163 kB 00:00 2023-12-02T22:19:00.249 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2023-12-02T22:19:00.300 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2023-12-02T22:19:00.317 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 935 kB/s | 63 kB 00:00 2023-12-02T22:19:00.503 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:00.583 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-02T22:19:00.627 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-02T22:19:00.660 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-02T22:19:00.703 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-02T22:19:00.730 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 5/6 2023-12-02T22:19:02.041 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): ceph-test-18.0.0-7670.g0c523c70.el9.x86_ 20 MB/s | 41 MB 00:02 2023-12-02T22:19:02.045 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:02.045 INFO:teuthology.orchestra.run.smithi012.stdout:Total 20 MB/s | 42 MB 00:02 2023-12-02T22:19:02.045 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:02.069 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:02.069 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:02.441 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:02.441 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:03.009 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:03.101 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-02T22:19:03.141 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-02T22:19:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-02T22:19:03.223 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-02T22:19:03.245 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 5/6 2023-12-02T22:19:04.613 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 5/6 2023-12-02T22:19:04.688 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 6/6 2023-12-02T22:19:06.980 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 6/6 2023-12-02T22:19:06.980 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 1/6 2023-12-02T22:19:06.980 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 2/6 2023-12-02T22:19:06.980 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-02T22:19:06.980 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-02T22:19:06.981 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-02T22:19:07.296 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 5/6 2023-12-02T22:19:07.325 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-02T22:19:07.325 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:07.325 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:07.325 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:07.325 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2023-12-02T22:19:07.326 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:07.326 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-02T22:19:07.326 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-02T22:19:07.326 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-02T22:19:07.326 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:07.326 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:07.378 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 6/6 2023-12-02T22:19:07.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2023-12-02T22:19:08.204 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:39 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:08.353 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 6.6 k 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:19:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 2.1 M 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 1.5 M 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 243 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 4.4 M 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 18 M 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-02T22:19:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-02T22:19:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout:Install 36 Packages 2023-12-02T22:19:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:08.360 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 30 M 2023-12-02T22:19:08.360 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 111 M 2023-12-02T22:19:08.360 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:08.971 INFO:teuthology.orchestra.run.smithi170.stdout:(1/36): ceph-18.0.0-7670.g0c523c70.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2023-12-02T22:19:09.221 INFO:teuthology.orchestra.run.smithi170.stdout:(2/36): ceph-mgr-18.0.0-7670.g0c523c70.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2023-12-02T22:19:09.499 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 6/6 2023-12-02T22:19:09.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 1/6 2023-12-02T22:19:09.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 2/6 2023-12-02T22:19:09.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-02T22:19:09.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-02T22:19:09.499 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-02T22:19:09.513 INFO:teuthology.orchestra.run.smithi170.stdout:(3/36): ceph-mon-18.0.0-7670.g0c523c70.el9.x86_ 8.1 MB/s | 4.4 MB 00:00 2023-12-02T22:19:09.605 INFO:teuthology.orchestra.run.smithi170.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7670.g0c52 2.6 MB/s | 243 kB 00:00 2023-12-02T22:19:09.705 INFO:teuthology.orchestra.run.smithi170.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-12-02T22:19:09.739 INFO:teuthology.orchestra.run.smithi170.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2023-12-02T22:19:09.781 INFO:teuthology.orchestra.run.smithi170.stdout:(7/36): ceph-mds-18.0.0-7670.g0c523c70.el9.x86_ 2.1 MB/s | 2.1 MB 00:00 2023-12-02T22:19:09.797 INFO:teuthology.orchestra.run.smithi170.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 539 kB/s | 32 kB 00:00 2023-12-02T22:19:09.814 INFO:teuthology.orchestra.run.smithi170.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-02T22:19:09.831 INFO:teuthology.orchestra.run.smithi170.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-02T22:19:09.840 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-02T22:19:09.840 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:09.840 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:09.841 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:09.858 INFO:teuthology.orchestra.run.smithi170.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-02T22:19:09.907 INFO:teuthology.orchestra.run.smithi170.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-02T22:19:09.928 INFO:teuthology.orchestra.run.smithi170.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-12-02T22:19:09.952 INFO:teuthology.orchestra.run.smithi170.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.7 MB/s | 124 kB 00:00 2023-12-02T22:19:09.978 INFO:teuthology.orchestra.run.smithi170.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-02T22:19:10.003 INFO:teuthology.orchestra.run.smithi170.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-02T22:19:10.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph 2023-12-02T22:19:10.187 INFO:teuthology.orchestra.run.smithi170.stdout:(17/36): ceph-osd-18.0.0-7670.g0c523c70.el9.x86 18 MB/s | 18 MB 00:00 2023-12-02T22:19:10.238 INFO:teuthology.orchestra.run.smithi170.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2023-12-02T22:19:10.379 INFO:teuthology.orchestra.run.smithi170.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 402 kB/s | 151 kB 00:00 2023-12-02T22:19:10.404 INFO:teuthology.orchestra.run.smithi170.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 225 kB/s | 43 kB 00:00 2023-12-02T22:19:10.454 INFO:teuthology.orchestra.run.smithi170.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2023-12-02T22:19:10.484 INFO:teuthology.orchestra.run.smithi170.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 794 kB/s | 172 kB 00:00 2023-12-02T22:19:10.505 INFO:teuthology.orchestra.run.smithi170.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-12-02T22:19:10.530 INFO:teuthology.orchestra.run.smithi170.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2023-12-02T22:19:10.556 INFO:teuthology.orchestra.run.smithi170.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2023-12-02T22:19:10.581 INFO:teuthology.orchestra.run.smithi170.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2023-12-02T22:19:10.598 INFO:teuthology.orchestra.run.smithi170.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 292 kB/s | 20 kB 00:00 2023-12-02T22:19:10.615 INFO:teuthology.orchestra.run.smithi170.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2023-12-02T22:19:10.649 INFO:teuthology.orchestra.run.smithi170.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2023-12-02T22:19:10.665 INFO:teuthology.orchestra.run.smithi170.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 325 kB/s | 16 kB 00:00 2023-12-02T22:19:10.699 INFO:teuthology.orchestra.run.smithi170.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2023-12-02T22:19:10.725 INFO:teuthology.orchestra.run.smithi170.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2023-12-02T22:19:10.734 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:40 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:10.741 INFO:teuthology.orchestra.run.smithi170.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2023-12-02T22:19:10.778 INFO:teuthology.orchestra.run.smithi170.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 596 kB/s | 20 kB 00:00 2023-12-02T22:19:10.801 INFO:teuthology.orchestra.run.smithi170.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-02T22:19:10.875 INFO:teuthology.orchestra.run.smithi170.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2023-12-02T22:19:10.883 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:10.883 INFO:teuthology.orchestra.run.smithi170.stdout:Total 12 MB/s | 30 MB 00:02 2023-12-02T22:19:10.894 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout:========================================================================================= 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout:========================================================================================= 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 6.6 k 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 2.1 M 2023-12-02T22:19:10.895 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 1.5 M 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 243 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 4.4 M 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 18 M 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-02T22:19:10.896 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-02T22:19:10.897 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout:========================================================================================= 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout:Install 36 Packages 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 30 M 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 111 M 2023-12-02T22:19:10.898 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:10.930 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:10.930 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:11.327 INFO:teuthology.orchestra.run.smithi012.stdout:(1/36): ceph-18.0.0-7670.g0c523c70.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2023-12-02T22:19:11.360 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:11.360 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:11.635 INFO:teuthology.orchestra.run.smithi012.stdout:(2/36): ceph-mds-18.0.0-7670.g0c523c70.el9.x86_ 4.0 MB/s | 2.1 MB 00:00 2023-12-02T22:19:11.694 INFO:teuthology.orchestra.run.smithi012.stdout:(3/36): ceph-mgr-18.0.0-7670.g0c523c70.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2023-12-02T22:19:11.913 INFO:teuthology.orchestra.run.smithi012.stdout:(4/36): ceph-mon-18.0.0-7670.g0c523c70.el9.x86_ 7.5 MB/s | 4.4 MB 00:00 2023-12-02T22:19:12.027 INFO:teuthology.orchestra.run.smithi012.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-02T22:19:12.053 INFO:teuthology.orchestra.run.smithi012.stdout:(6/36): ceph-mgr-modules-core-18.0.0-7670.g0c52 678 kB/s | 243 kB 00:00 2023-12-02T22:19:12.070 INFO:teuthology.orchestra.run.smithi012.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-02T22:19:12.099 INFO:teuthology.orchestra.run.smithi012.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 762 kB/s | 32 kB 00:00 2023-12-02T22:19:12.120 INFO:teuthology.orchestra.run.smithi012.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2023-12-02T22:19:12.146 INFO:teuthology.orchestra.run.smithi012.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2023-12-02T22:19:12.171 INFO:teuthology.orchestra.run.smithi012.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-02T22:19:12.246 INFO:teuthology.orchestra.run.smithi012.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2023-12-02T22:19:12.271 INFO:teuthology.orchestra.run.smithi012.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2023-12-02T22:19:12.296 INFO:teuthology.orchestra.run.smithi012.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-02T22:19:12.322 INFO:teuthology.orchestra.run.smithi012.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-02T22:19:12.348 INFO:teuthology.orchestra.run.smithi012.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-02T22:19:12.497 INFO:teuthology.orchestra.run.smithi012.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2023-12-02T22:19:12.673 INFO:teuthology.orchestra.run.smithi012.stdout:(18/36): ceph-osd-18.0.0-7670.g0c523c70.el9.x86 17 MB/s | 18 MB 00:01 2023-12-02T22:19:12.706 INFO:teuthology.orchestra.run.smithi012.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 421 kB/s | 151 kB 00:00 2023-12-02T22:19:12.723 INFO:teuthology.orchestra.run.smithi012.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 193 kB/s | 43 kB 00:00 2023-12-02T22:19:12.734 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:12.748 INFO:teuthology.orchestra.run.smithi012.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2023-12-02T22:19:12.790 INFO:teuthology.orchestra.run.smithi012.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2023-12-02T22:19:12.795 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-02T22:19:12.807 INFO:teuthology.orchestra.run.smithi012.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2023-12-02T22:19:12.824 INFO:teuthology.orchestra.run.smithi012.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 236 kB/s | 18 kB 00:00 2023-12-02T22:19:12.827 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-02T22:19:12.841 INFO:teuthology.orchestra.run.smithi012.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 692 kB/s | 23 kB 00:00 2023-12-02T22:19:12.856 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-02T22:19:12.858 INFO:teuthology.orchestra.run.smithi012.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2023-12-02T22:19:12.875 INFO:teuthology.orchestra.run.smithi012.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2023-12-02T22:19:12.892 INFO:teuthology.orchestra.run.smithi012.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2023-12-02T22:19:12.908 INFO:teuthology.orchestra.run.smithi012.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-12-02T22:19:12.925 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-02T22:19:12.960 INFO:teuthology.orchestra.run.smithi012.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2023-12-02T22:19:12.971 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-02T22:19:12.976 INFO:teuthology.orchestra.run.smithi012.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2023-12-02T22:19:13.001 INFO:teuthology.orchestra.run.smithi012.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 982 kB/s | 90 kB 00:00 2023-12-02T22:19:13.006 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-02T22:19:13.026 INFO:teuthology.orchestra.run.smithi012.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2023-12-02T22:19:13.052 INFO:teuthology.orchestra.run.smithi012.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-12-02T22:19:13.077 INFO:teuthology.orchestra.run.smithi012.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2023-12-02T22:19:13.095 INFO:teuthology.orchestra.run.smithi012.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 305 kB/s | 20 kB 00:00 2023-12-02T22:19:13.095 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-02T22:19:13.098 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:13.098 INFO:teuthology.orchestra.run.smithi012.stdout:Total 14 MB/s | 30 MB 00:02 2023-12-02T22:19:13.111 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:13.160 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:13.160 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:13.186 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-02T22:19:13.225 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-02T22:19:13.294 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-02T22:19:13.357 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-02T22:19:13.395 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-02T22:19:13.445 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-02T22:19:13.510 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-02T22:19:13.610 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:13.610 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:13.702 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-02T22:19:13.750 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-02T22:19:13.781 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-02T22:19:13.815 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-02T22:19:13.845 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-02T22:19:13.921 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-02T22:19:13.955 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T22:19:13.991 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-02T22:19:14.029 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-02T22:19:14.065 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-02T22:19:14.100 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-02T22:19:14.188 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-02T22:19:14.238 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-02T22:19:14.310 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-02T22:19:14.446 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-02T22:19:14.563 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T22:19:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7670.g0c523c70.el9. 31/36 2023-12-02T22:19:14.752 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 32/36 2023-12-02T22:19:14.778 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 32/36 2023-12-02T22:19:14.778 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:14.779 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-02T22:19:14.779 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T22:19:14.779 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T22:19:14.781 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:14.968 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:15.021 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-02T22:19:15.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-02T22:19:15.082 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-02T22:19:15.156 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-02T22:19:15.198 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-02T22:19:15.232 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-02T22:19:15.326 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-02T22:19:15.421 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-02T22:19:15.468 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-02T22:19:15.541 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-02T22:19:15.593 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-02T22:19:15.639 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-02T22:19:15.697 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-02T22:19:15.768 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-02T22:19:15.978 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-02T22:19:16.027 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-02T22:19:16.064 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-02T22:19:16.099 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-02T22:19:16.129 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-02T22:19:16.208 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-02T22:19:16.240 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T22:19:16.273 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-02T22:19:16.314 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-02T22:19:16.358 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-02T22:19:16.391 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-02T22:19:16.419 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 33/36 2023-12-02T22:19:16.446 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 33/36 2023-12-02T22:19:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-02T22:19:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T22:19:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T22:19:16.446 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:16.487 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-02T22:19:16.531 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-02T22:19:16.605 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-02T22:19:16.755 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-02T22:19:16.878 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T22:19:16.895 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 34/36 2023-12-02T22:19:16.918 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 34/36 2023-12-02T22:19:16.918 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:16.918 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-02T22:19:16.918 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T22:19:16.919 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T22:19:16.919 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:17.033 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7670.g0c523c70.el9. 31/36 2023-12-02T22:19:17.060 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 32/36 2023-12-02T22:19:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 32/36 2023-12-02T22:19:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-02T22:19:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T22:19:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T22:19:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:17.112 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 35/36 2023-12-02T22:19:17.135 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 35/36 2023-12-02T22:19:17.136 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:17.136 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-02T22:19:17.136 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T22:19:17.136 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T22:19:17.136 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:17.193 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 36/36 2023-12-02T22:19:18.720 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 33/36 2023-12-02T22:19:18.748 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 33/36 2023-12-02T22:19:18.749 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:18.749 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-02T22:19:18.749 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T22:19:18.749 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T22:19:18.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:18.760 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 36/36 2023-12-02T22:19:18.761 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 1/36 2023-12-02T22:19:18.761 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 2/36 2023-12-02T22:19:18.761 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 3/36 2023-12-02T22:19:18.761 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 4/36 2023-12-02T22:19:18.761 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 5/36 2023-12-02T22:19:18.761 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7670.g0c523c70.el9. 6/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-02T22:19:18.762 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-02T22:19:18.763 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-02T22:19:18.764 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-02T22:19:19.220 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 34/36 2023-12-02T22:19:19.244 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 34/36 2023-12-02T22:19:19.244 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:19.244 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-02T22:19:19.244 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T22:19:19.244 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T22:19:19.245 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:19.454 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 35/36 2023-12-02T22:19:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 35/36 2023-12-02T22:19:19.478 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:19.478 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-02T22:19:19.478 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T22:19:19.478 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T22:19:19.478 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:19.545 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 36/36 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:19.837 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-02T22:19:19.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-02T22:19:19.839 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-02T22:19:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-02T22:19:19.841 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-02T22:19:19.841 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:19.841 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:20.213 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2023-12-02T22:19:20.793 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:51 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:20.892 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:19:20.942 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:20.946 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:19:20.946 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:21.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2023-12-02T22:19:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 36/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 1/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 2/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 3/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 4/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 5/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7670.g0c523c70.el9. 6/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-02T22:19:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-02T22:19:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-02T22:19:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-02T22:19:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-02T22:19:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-02T22:19:21.586 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:52 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 776 k 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:21.732 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:21.733 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:21.733 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:19:21.733 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:21.733 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 776 k 2023-12-02T22:19:21.733 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 790 k 2023-12-02T22:19:21.733 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:22.095 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-18.0.0-7670.g0c523c70.el9.noarch.rpm 2.1 MB/s | 776 kB 00:00 2023-12-02T22:19:22.096 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:22.096 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.1 MB/s | 776 kB 00:00 2023-12-02T22:19:22.096 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:22.101 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:22.101 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:22.107 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:22.107 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:22.184 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:22.356 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-02T22:19:22.357 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-02T22:19:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-02T22:19:22.359 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-02T22:19:22.359 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-02T22:19:22.359 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-02T22:19:22.359 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-02T22:19:22.359 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:22.359 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:22.600 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:22.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-base 2023-12-02T22:19:22.699 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:23.107 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:23.288 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:53 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:23.387 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-base-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:19:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:23.417 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:23.417 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:23.439 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:23.440 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:19:23.440 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:23.507 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install cephadm 2023-12-02T22:19:23.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2023-12-02T22:19:24.109 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:54 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:24.192 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:55 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 776 k 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:24.260 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:24.261 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:19:24.261 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:24.261 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 776 k 2023-12-02T22:19:24.261 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 790 k 2023-12-02T22:19:24.261 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:24.338 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 141 k 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:19:24.339 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:24.340 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 141 k 2023-12-02T22:19:24.340 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 396 k 2023-12-02T22:19:24.340 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:24.560 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-18.0.0-7670.g0c523c 644 kB/s | 141 kB 00:00 2023-12-02T22:19:24.560 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:24.560 INFO:teuthology.orchestra.run.smithi170.stdout:Total 640 kB/s | 141 kB 00:00 2023-12-02T22:19:24.561 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:24.567 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:24.567 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:24.619 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:24.620 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:24.636 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm-18.0.0-7670.g0c523c70.el9.noarch.rpm 2.0 MB/s | 776 kB 00:00 2023-12-02T22:19:24.637 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:24.637 INFO:teuthology.orchestra.run.smithi012.stdout:Total 2.0 MB/s | 776 kB 00:00 2023-12-02T22:19:24.637 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:24.643 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:24.643 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:24.649 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:24.649 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:24.729 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70. 1/1 2023-12-02T22:19:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70. 1/1 2023-12-02T22:19:24.843 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:24.844 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-02T22:19:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:25.144 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:25.225 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:25.591 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70. 1/1 2023-12-02T22:19:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:25.829 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:25.829 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:25.829 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:25.890 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:25.890 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:25.891 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:25.891 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:25.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:25.891 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:26.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2023-12-02T22:19:26.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-immutable-object-cache 2023-12-02T22:19:26.647 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:57 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:26.692 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:56 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:26.742 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:19:26.790 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:26.792 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:19:26.792 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:26.845 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:26.845 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:26.845 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 141 k 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 141 k 2023-12-02T22:19:26.846 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 396 k 2023-12-02T22:19:26.847 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:26.856 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2023-12-02T22:19:27.200 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-immutable-object-cache-18.0.0-7670.g0c523c 399 kB/s | 141 kB 00:00 2023-12-02T22:19:27.200 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:27.201 INFO:teuthology.orchestra.run.smithi012.stdout:Total 397 kB/s | 141 kB 00:00 2023-12-02T22:19:27.201 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:27.209 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:27.209 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:27.264 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:27.265 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:27.414 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:27.429 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:58 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70. 1/1 2023-12-02T22:19:27.502 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70. 1/1 2023-12-02T22:19:27.502 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:27.502 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-02T22:19:27.502 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:27.577 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:27.577 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:27.577 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:27.577 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:27.577 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:27.577 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 1.8 M 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 28 k 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 15 k 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout:Install 7 Packages 2023-12-02T22:19:27.578 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:27.579 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.1 M 2023-12-02T22:19:27.579 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 8.7 M 2023-12-02T22:19:27.579 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:28.204 INFO:teuthology.orchestra.run.smithi170.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7670.g0c5 108 kB/s | 28 kB 00:00 2023-12-02T22:19:28.222 INFO:teuthology.orchestra.run.smithi170.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7670.g0c52 54 kB/s | 15 kB 00:00 2023-12-02T22:19:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70. 1/1 2023-12-02T22:19:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:28.592 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:19:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:28.593 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:28.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr 2023-12-02T22:19:28.905 INFO:teuthology.orchestra.run.smithi170.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7670.g0c523c70 1.8 MB/s | 1.8 MB 00:00 2023-12-02T22:19:29.410 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:59 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:29.509 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-mgr-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:19:29.561 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:29.562 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:19:29.563 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:29.624 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-dashboard 2023-12-02T22:19:30.223 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:00 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 1.8 M 2023-12-02T22:19:30.380 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 28 k 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 15 k 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout:Install 7 Packages 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 2.1 M 2023-12-02T22:19:30.381 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 8.7 M 2023-12-02T22:19:30.382 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:30.728 INFO:teuthology.orchestra.run.smithi012.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7670.g0c52 58 kB/s | 15 kB 00:00 2023-12-02T22:19:30.762 INFO:teuthology.orchestra.run.smithi012.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7670.g0c5 95 kB/s | 28 kB 00:00 2023-12-02T22:19:30.813 INFO:teuthology.orchestra.run.smithi170.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 3.5 kB/s | 9.0 kB 00:02 2023-12-02T22:19:30.880 INFO:teuthology.orchestra.run.smithi170.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 15 kB/s | 41 kB 00:02 2023-12-02T22:19:31.050 INFO:teuthology.orchestra.run.smithi170.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 808 kB/s | 188 kB 00:00 2023-12-02T22:19:31.062 INFO:teuthology.orchestra.run.smithi012.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7670.g0c523c70 3.0 MB/s | 1.8 MB 00:00 2023-12-02T22:19:31.113 INFO:teuthology.orchestra.run.smithi012.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 24 kB/s | 9.0 kB 00:00 2023-12-02T22:19:31.296 INFO:teuthology.orchestra.run.smithi012.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 77 kB/s | 41 kB 00:00 2023-12-02T22:19:31.521 INFO:teuthology.orchestra.run.smithi012.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 67 kB/s | 31 kB 00:00 2023-12-02T22:19:31.896 INFO:teuthology.orchestra.run.smithi012.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 241 kB/s | 188 kB 00:00 2023-12-02T22:19:31.900 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:31.900 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.4 MB/s | 2.1 MB 00:01 2023-12-02T22:19:31.900 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:31.910 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:31.910 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:31.989 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:31.990 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:32.123 INFO:teuthology.orchestra.run.smithi170.stdout:(7/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 9.5 kB/s | 31 kB 00:03 2023-12-02T22:19:32.124 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:32.126 INFO:teuthology.orchestra.run.smithi170.stdout:Total 465 kB/s | 2.1 MB 00:04 2023-12-02T22:19:32.126 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:32.134 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:32.135 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:32.209 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:32.209 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:32.425 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:32.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-02T22:19:32.515 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:32.532 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-02T22:19:32.565 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-02T22:19:32.580 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-02T22:19:32.599 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-02T22:19:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-02T22:19:32.640 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7670.g0c523c70.el9.n 5/7 2023-12-02T22:19:32.655 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-02T22:19:32.694 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-02T22:19:32.724 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7670.g0c523c70.el9.n 5/7 2023-12-02T22:19:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7670.g0c523c70.el9. 6/7 2023-12-02T22:19:32.925 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarc 7/7 2023-12-02T22:19:32.961 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7670.g0c523c70.el9. 6/7 2023-12-02T22:19:32.993 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarc 7/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarc 7/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7670.g0c523c70.el9. 1/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarc 2/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7670.g0c523c70.el9.n 3/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-02T22:19:33.466 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-02T22:19:33.471 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarc 7/7 2023-12-02T22:19:33.471 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7670.g0c523c70.el9. 1/7 2023-12-02T22:19:33.471 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarc 2/7 2023-12-02T22:19:33.471 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7670.g0c523c70.el9.n 3/7 2023-12-02T22:19:33.471 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-02T22:19:33.471 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-02T22:19:33.472 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-02T22:19:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-02T22:19:33.777 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-02T22:19:33.777 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-02T22:19:33.777 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:33.777 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:33.802 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-02T22:19:33.802 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:33.802 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:33.802 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:33.802 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:33.802 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:33.803 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-02T22:19:33.803 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-02T22:19:33.803 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-02T22:19:33.803 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-02T22:19:33.803 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:33.803 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:34.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-02T22:19:34.015 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-02T22:19:34.578 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:05 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:34.614 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:04 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:34.722 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:34.723 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 7.4 M 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-02T22:19:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:34.725 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2023-12-02T22:19:34.726 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 209 M 2023-12-02T22:19:34.726 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:34.765 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:34.766 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-12-02T22:19:34.766 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:34.766 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 7.4 M 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-02T22:19:34.767 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-02T22:19:34.768 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-02T22:19:34.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-02T22:19:34.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-02T22:19:34.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-02T22:19:34.769 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-02T22:19:34.769 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:34.769 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:34.769 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-12-02T22:19:34.769 INFO:teuthology.orchestra.run.smithi012.stdout:Install 12 Packages 2023-12-02T22:19:34.769 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:34.778 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 37 M 2023-12-02T22:19:34.778 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 209 M 2023-12-02T22:19:34.778 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:34.938 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2023-12-02T22:19:34.982 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 427 kB/s | 17 kB 00:00 2023-12-02T22:19:35.013 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-02T22:19:35.063 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-02T22:19:35.278 INFO:teuthology.orchestra.run.smithi012.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 159 kB/s | 32 kB 00:00 2023-12-02T22:19:35.314 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-12-02T22:19:35.346 INFO:teuthology.orchestra.run.smithi012.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 257 kB/s | 17 kB 00:00 2023-12-02T22:19:35.389 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2023-12-02T22:19:35.394 INFO:teuthology.orchestra.run.smithi012.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2023-12-02T22:19:35.406 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-02T22:19:35.677 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2023-12-02T22:19:35.795 INFO:teuthology.orchestra.run.smithi012.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.2 MB/s | 3.0 MB 00:00 2023-12-02T22:19:35.950 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2023-12-02T22:19:35.974 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 661 kB/s | 195 kB 00:00 2023-12-02T22:19:36.149 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2023-12-02T22:19:36.479 INFO:teuthology.orchestra.run.smithi012.stdout:(5/12): ceph-mgr-diskprediction-local-18.0.0-76 5.3 MB/s | 7.4 MB 00:01 2023-12-02T22:19:36.787 INFO:teuthology.orchestra.run.smithi012.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 707 kB/s | 217 kB 00:00 2023-12-02T22:19:37.137 INFO:teuthology.orchestra.run.smithi012.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 2.7 MB/s | 4.7 MB 00:01 2023-12-02T22:19:37.307 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 2.9 MB/s | 7.4 MB 00:02 2023-12-02T22:19:37.313 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:37.313 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 37 MB 00:02 2023-12-02T22:19:37.325 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:37.352 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:37.352 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:37.612 INFO:teuthology.orchestra.run.smithi012.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.6 MB/s | 4.8 MB 00:01 2023-12-02T22:19:37.638 INFO:teuthology.orchestra.run.smithi012.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2023-12-02T22:19:37.733 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:37.733 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:37.896 INFO:teuthology.orchestra.run.smithi012.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 688 kB/s | 195 kB 00:00 2023-12-02T22:19:37.955 INFO:teuthology.orchestra.run.smithi012.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 779 kB/s | 245 kB 00:00 2023-12-02T22:19:38.442 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:38.531 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-02T22:19:38.561 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-02T22:19:38.613 INFO:teuthology.orchestra.run.smithi012.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.6 MB/s | 16 MB 00:01 2023-12-02T22:19:38.616 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:38.616 INFO:teuthology.orchestra.run.smithi012.stdout:Total 9.6 MB/s | 37 MB 00:03 2023-12-02T22:19:38.631 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:38.646 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-02T22:19:38.658 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:38.658 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:38.681 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-02T22:19:38.850 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T22:19:38.880 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T22:19:38.960 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-02T22:19:39.070 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:39.070 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:39.626 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-02T22:19:39.736 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-02T22:19:39.792 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:39.892 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-02T22:19:39.926 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-02T22:19:40.025 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-02T22:19:40.064 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-02T22:19:40.240 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T22:19:40.262 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T22:19:40.345 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-02T22:19:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-02T22:19:41.053 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-02T22:19:41.169 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-02T22:19:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-02T22:19:42.039 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523 12/12 2023-12-02T22:19:42.475 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-02T22:19:43.546 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-02T22:19:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523 12/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523 12/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523 1/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-02T22:19:44.581 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-02T22:19:44.997 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-02T22:19:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-02T22:19:44.999 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-02T22:19:44.999 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-02T22:19:44.999 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-02T22:19:44.999 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:44.999 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:45.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2023-12-02T22:19:45.989 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:16 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523 12/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523 1/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T22:19:46.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-02T22:19:46.059 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-02T22:19:46.059 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-02T22:19:46.059 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-02T22:19:46.059 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-02T22:19:46.149 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:46.150 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-12-02T22:19:46.150 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 50 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-02T22:19:46.151 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:46.152 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.0 M 2023-12-02T22:19:46.153 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2023-12-02T22:19:46.153 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:46.350 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 214 kB/s | 24 kB 00:00 2023-12-02T22:19:46.367 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2023-12-02T22:19:46.426 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-02T22:19:46.450 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2023-12-02T22:19:46.468 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-02T22:19:46.468 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:46.468 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-02T22:19:46.469 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-02T22:19:46.470 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:46.470 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:46.476 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-02T22:19:46.493 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2023-12-02T22:19:46.535 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 359 kB/s | 15 kB 00:00 2023-12-02T22:19:46.560 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 384 kB/s | 32 kB 00:00 2023-12-02T22:19:46.585 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): ceph-mgr-rook-18.0.0-7670.g0c523c70.el9 144 kB/s | 50 kB 00:00 2023-12-02T22:19:46.636 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 1.9 MB/s | 197 kB 00:00 2023-12-02T22:19:46.661 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2023-12-02T22:19:46.686 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2023-12-02T22:19:46.728 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2023-12-02T22:19:46.730 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:46.730 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2023-12-02T22:19:46.730 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:46.743 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:46.744 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:46.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-rook 2023-12-02T22:19:46.840 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:46.840 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:47.353 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:47.430 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:17 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:47.526 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-02T22:19:47.579 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-02T22:19:47.594 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 50 k 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-02T22:19:47.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout:Install 13 Packages 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 2.0 M 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 26 M 2023-12-02T22:19:47.596 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:47.642 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-02T22:19:47.688 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-02T22:19:47.730 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-02T22:19:47.793 INFO:teuthology.orchestra.run.smithi012.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 158 kB/s | 18 kB 00:00 2023-12-02T22:19:47.818 INFO:teuthology.orchestra.run.smithi012.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 24 kB 00:00 2023-12-02T22:19:47.822 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-02T22:19:47.859 INFO:teuthology.orchestra.run.smithi012.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-02T22:19:47.940 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-02T22:19:47.998 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-02T22:19:48.005 INFO:teuthology.orchestra.run.smithi012.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 726 kB/s | 132 kB 00:00 2023-12-02T22:19:48.027 INFO:teuthology.orchestra.run.smithi012.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 211 kB 00:00 2023-12-02T22:19:48.052 INFO:teuthology.orchestra.run.smithi012.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-12-02T22:19:48.077 INFO:teuthology.orchestra.run.smithi012.stdout:(7/13): ceph-mgr-rook-18.0.0-7670.g0c523c70.el9 126 kB/s | 50 kB 00:00 2023-12-02T22:19:48.444 INFO:teuthology.orchestra.run.smithi012.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2023-12-02T22:19:48.469 INFO:teuthology.orchestra.run.smithi012.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 73 kB/s | 32 kB 00:00 2023-12-02T22:19:48.686 INFO:teuthology.orchestra.run.smithi012.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 324 kB/s | 197 kB 00:00 2023-12-02T22:19:48.726 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-02T22:19:48.779 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-02T22:19:48.812 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-02T22:19:48.820 INFO:teuthology.orchestra.run.smithi012.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2023-12-02T22:19:48.863 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-02T22:19:48.882 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 13/13 2023-12-02T22:19:48.945 INFO:teuthology.orchestra.run.smithi012.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 347 kB/s | 90 kB 00:00 2023-12-02T22:19:48.979 INFO:teuthology.orchestra.run.smithi012.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-12-02T22:19:48.985 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:48.986 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-12-02T22:19:48.986 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:48.999 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:48.999 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:49.110 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:49.111 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 13/13 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 1/13 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-02T22:19:49.649 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-02T22:19:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-02T22:19:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-02T22:19:49.650 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-02T22:19:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-02T22:19:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-02T22:19:49.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-02T22:19:49.662 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:49.825 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-02T22:19:49.883 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-02T22:19:49.940 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-02T22:19:49.975 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-02T22:19:50.013 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-02T22:19:50.014 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-02T22:19:50.015 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-02T22:19:50.015 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-02T22:19:50.015 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:50.015 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:50.016 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-02T22:19:50.116 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-02T22:19:50.224 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-02T22:19:50.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2023-12-02T22:19:50.276 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-02T22:19:50.837 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:21 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:50.981 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:50.983 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-12-02T22:19:50.983 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:50.983 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-12-02T22:19:50.983 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:50.983 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 133 k 2023-12-02T22:19:50.983 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.8 M 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 33 M 2023-12-02T22:19:50.984 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:51.043 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-02T22:19:51.098 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-02T22:19:51.131 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-02T22:19:51.180 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-02T22:19:51.209 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 13/13 2023-12-02T22:19:51.297 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-02T22:19:51.356 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7670.g0c523c70.e 613 kB/s | 133 kB 00:00 2023-12-02T22:19:51.431 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-12-02T22:19:51.856 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 116 kB/s | 58 kB 00:00 2023-12-02T22:19:51.915 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 116 kB/s | 51 kB 00:00 2023-12-02T22:19:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 13/13 2023-12-02T22:19:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 1/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-02T22:19:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-02T22:19:52.059 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 723 kB/s | 548 kB 00:00 2023-12-02T22:19:52.064 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:52.064 INFO:teuthology.orchestra.run.smithi170.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2023-12-02T22:19:52.064 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:52.070 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:52.070 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:52.158 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:52.158 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-02T22:19:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-02T22:19:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-02T22:19:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-02T22:19:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:52.368 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:52.455 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:52.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-cephadm 2023-12-02T22:19:52.623 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-02T22:19:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-02T22:19:53.176 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-02T22:19:53.201 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:23 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:53.254 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-02T22:19:53.314 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-02T22:19:53.348 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 6/6 2023-12-02T22:19:53.353 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:53.353 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2023-12-02T22:19:53.353 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:53.353 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 133 k 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2023-12-02T22:19:53.354 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:53.355 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 6.8 M 2023-12-02T22:19:53.355 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 33 M 2023-12-02T22:19:53.355 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:53.765 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-02T22:19:53.840 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7670.g0c523c70.e 537 kB/s | 133 kB 00:00 2023-12-02T22:19:53.924 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-12-02T22:19:53.999 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2023-12-02T22:19:54.025 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2023-12-02T22:19:54.050 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2023-12-02T22:19:54.051 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:54.052 INFO:teuthology.orchestra.run.smithi012.stdout:Total 9.7 MB/s | 6.8 MB 00:00 2023-12-02T22:19:54.052 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:54.062 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:54.062 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:54.110 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 6/6 2023-12-02T22:19:54.110 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/6 2023-12-02T22:19:54.111 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-02T22:19:54.111 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-02T22:19:54.111 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-02T22:19:54.111 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-02T22:19:54.157 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:54.157 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:54.406 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:54.452 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:54.621 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-02T22:19:54.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2023-12-02T22:19:54.677 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-02T22:19:55.228 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-02T22:19:55.247 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:26 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:55.314 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-02T22:19:55.374 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-02T22:19:55.392 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 860 k 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:55.393 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:55.394 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:55.394 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2023-12-02T22:19:55.394 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:55.394 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 939 k 2023-12-02T22:19:55.394 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.7 M 2023-12-02T22:19:55.394 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:55.396 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 6/6 2023-12-02T22:19:55.511 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 675 kB/s | 79 kB 00:00 2023-12-02T22:19:55.712 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-18.0.0-7670.g0c523c70.el9.x86_ 2.7 MB/s | 860 kB 00:00 2023-12-02T22:19:55.712 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:55.712 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.9 MB/s | 939 kB 00:00 2023-12-02T22:19:55.713 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:55.732 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:55.732 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:55.807 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:55.807 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:55.987 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:56.118 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-02T22:19:56.185 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 2/2 2023-12-02T22:19:56.247 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 6/6 2023-12-02T22:19:56.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 1/6 2023-12-02T22:19:56.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-02T22:19:56.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-02T22:19:56.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-02T22:19:56.247 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:56.577 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:56.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-fuse 2023-12-02T22:19:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 2/2 2023-12-02T22:19:56.910 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/2 2023-12-02T22:19:57.220 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-02T22:19:57.220 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:57.220 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:57.220 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-02T22:19:57.220 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:57.221 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:19:57.402 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:27 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:19:57.476 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-volume 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 860 k 2023-12-02T22:19:57.554 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout:Install 2 Packages 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:57.555 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 939 k 2023-12-02T22:19:57.556 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 2.7 M 2023-12-02T22:19:57.556 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:19:57.701 INFO:teuthology.orchestra.run.smithi012.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 543 kB/s | 79 kB 00:00 2023-12-02T22:19:58.061 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:29 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:19:58.109 INFO:teuthology.orchestra.run.smithi012.stdout:(2/2): ceph-fuse-18.0.0-7670.g0c523c70.el9.x86_ 1.5 MB/s | 860 kB 00:00 2023-12-02T22:19:58.111 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:58.111 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.6 MB/s | 939 kB 00:00 2023-12-02T22:19:58.111 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:19:58.134 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:19:58.134 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:19:58.211 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 262 k 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:19:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:58.213 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 262 k 2023-12-02T22:19:58.213 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.2 M 2023-12-02T22:19:58.213 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:19:58.215 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:19:58.215 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:19:58.399 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:19:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-02T22:19:58.595 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 2/2 2023-12-02T22:19:58.599 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-volume-18.0.0-7670.g0c523c70.el9.noarch.rp 679 kB/s | 262 kB 00:00 2023-12-02T22:19:58.600 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:19:58.600 INFO:teuthology.orchestra.run.smithi170.stdout:Total 676 kB/s | 262 kB 00:00 2023-12-02T22:19:58.600 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:19:58.609 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:19:58.609 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:19:58.645 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:19:58.646 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:19:58.862 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:19:58.927 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:58.942 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:58.942 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:19:58.942 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-02T22:19:58.942 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 2/2 2023-12-02T22:19:59.277 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/2 2023-12-02T22:19:59.608 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-02T22:19:59.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:59.609 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:19:59.609 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-02T22:19:59.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:19:59.609 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:19:59.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-volume 2023-12-02T22:19:59.981 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:19:59.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:59.981 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:19:59.981 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:19:59.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:19:59.981 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:00.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2023-12-02T22:20:00.458 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:30 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:00.609 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:18.0.0-7670.g0c523c70.el9 ceph-noarch 262 k 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:20:00.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:00.611 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 262 k 2023-12-02T22:20:00.611 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 1.2 M 2023-12-02T22:20:00.611 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:20:00.871 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:31 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:01.005 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-volume-18.0.0-7670.g0c523c70.el9.noarch.rp 666 kB/s | 262 kB 00:00 2023-12-02T22:20:01.005 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:01.006 INFO:teuthology.orchestra.run.smithi012.stdout:Total 663 kB/s | 262 kB 00:00 2023-12-02T22:20:01.006 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:20:01.015 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:20:01.015 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 126 k 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:20:01.019 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:01.020 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 126 k 2023-12-02T22:20:01.020 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 456 k 2023-12-02T22:20:01.021 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:20:01.056 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:20:01.057 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:20:01.282 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:20:01.294 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-18.0.0-7670.g0c523c70.el9.x86_64 469 kB/s | 126 kB 00:00 2023-12-02T22:20:01.294 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:01.294 INFO:teuthology.orchestra.run.smithi170.stdout:Total 466 kB/s | 126 kB 00:00 2023-12-02T22:20:01.294 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:20:01.297 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:20:01.297 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:20:01.351 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:20:01.352 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:20:01.357 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:20:01.374 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:20:01.374 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:20:01.374 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-02T22:20:01.374 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:01.516 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:20:01.621 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:02.113 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:02.430 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:02.430 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:02.431 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:20:02.431 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:02.431 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:02.431 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:02.451 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 1/1 2023-12-02T22:20:02.451 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:02.452 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:20:02.452 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:18.0.0-7670.g0c523c70.el9.noarch 2023-12-02T22:20:02.452 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:02.452 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:02.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2023-12-02T22:20:02.885 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados-devel 2023-12-02T22:20:03.262 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:34 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:03.360 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:03.411 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:03.411 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:03.411 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:03.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2023-12-02T22:20:03.496 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:33 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:03.647 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 126 k 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:20:03.648 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:03.649 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:20:03.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:03.649 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 126 k 2023-12-02T22:20:03.649 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 456 k 2023-12-02T22:20:03.649 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:20:03.868 INFO:teuthology.orchestra.run.smithi012.stdout:librados-devel-18.0.0-7670.g0c523c70.el9.x86_64 578 kB/s | 126 kB 00:00 2023-12-02T22:20:03.869 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:03.869 INFO:teuthology.orchestra.run.smithi012.stdout:Total 573 kB/s | 126 kB 00:00 2023-12-02T22:20:03.869 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:20:03.875 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:20:03.875 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:20:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:20:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:20:04.068 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:35 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:04.202 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:20:04.213 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 31 k 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:04.214 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 k 2023-12-02T22:20:04.215 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 139 k 2023-12-02T22:20:04.215 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:20:04.303 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:04.451 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-18.0.0-7670.g0c523c70.el9.x86_6 132 kB/s | 31 kB 00:00 2023-12-02T22:20:04.452 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:04.452 INFO:teuthology.orchestra.run.smithi170.stdout:Total 131 kB/s | 31 kB 00:00 2023-12-02T22:20:04.452 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:20:04.456 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:20:04.456 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:20:04.484 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:20:04.484 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:20:04.564 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:20:04.686 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:04.734 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:05.058 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:05.058 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:05.058 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:20:05.058 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:05.058 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:05.058 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:05.083 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:05.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs2 2023-12-02T22:20:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:05.342 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:20:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:05.342 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:05.342 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:05.559 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2023-12-02T22:20:05.922 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:35 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:06.020 INFO:teuthology.orchestra.run.smithi012.stdout:Package libcephfs2-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:06.073 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:06.074 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:06.074 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:06.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs-devel 2023-12-02T22:20:06.137 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:37 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:06.231 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:06.282 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:06.283 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:06.283 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:06.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2023-12-02T22:20:06.737 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:36 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:06.890 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:06.890 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:06.890 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 31 k 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 31 k 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 139 k 2023-12-02T22:20:06.891 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:20:06.938 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:37 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:07.032 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:07.084 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:07.084 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:07.085 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:07.138 INFO:teuthology.orchestra.run.smithi012.stdout:libcephfs-devel-18.0.0-7670.g0c523c70.el9.x86_6 126 kB/s | 31 kB 00:00 2023-12-02T22:20:07.138 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:07.139 INFO:teuthology.orchestra.run.smithi012.stdout:Total 125 kB/s | 31 kB 00:00 2023-12-02T22:20:07.139 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:20:07.142 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:20:07.142 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:20:07.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2023-12-02T22:20:07.171 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:20:07.172 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:20:07.265 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:20:07.381 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:07.742 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:38 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:07.748 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:07.844 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:07.890 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:07.891 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:07.891 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:07.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2023-12-02T22:20:08.042 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:08.042 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:08.043 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:20:08.043 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:08.043 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:08.043 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:08.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados2 2023-12-02T22:20:08.538 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:39 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:08.636 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:08.684 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:08.706 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:08.707 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:08.754 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2023-12-02T22:20:08.788 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:38 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:08.886 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:08.939 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:08.941 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:08.941 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:09.002 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librbd1 2023-12-02T22:20:09.334 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:40 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:09.429 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:09.480 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:09.481 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:09.481 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:09.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2023-12-02T22:20:09.600 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:39 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:09.699 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:09.749 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:09.750 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:09.750 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:09.818 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rados 2023-12-02T22:20:10.137 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:41 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:10.232 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:10.282 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:10.283 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:10.283 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:10.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2023-12-02T22:20:10.425 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:40 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:10.524 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rados-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:10.577 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:10.578 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:10.578 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:10.643 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rgw 2023-12-02T22:20:10.935 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:41 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:11.076 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 85 k 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 85 k 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 231 k 2023-12-02T22:20:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:20:11.253 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:41 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:11.354 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rgw-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:11.387 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-18.0.0-7670.g0c523c70.el9.x86_64.rpm 275 kB/s | 85 kB 00:00 2023-12-02T22:20:11.388 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:11.388 INFO:teuthology.orchestra.run.smithi170.stdout:Total 273 kB/s | 85 kB 00:00 2023-12-02T22:20:11.388 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:20:11.395 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:20:11.396 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:20:11.404 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:11.405 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:11.406 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:11.450 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:20:11.450 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:20:11.468 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-cephfs 2023-12-02T22:20:11.583 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:20:11.677 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:12.077 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:42 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:12.100 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:12.176 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-cephfs-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:12.230 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:12.231 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:12.231 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:12.301 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rbd 2023-12-02T22:20:12.383 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:12.384 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:12.384 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:20:12.384 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:12.384 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:12.384 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:12.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2023-12-02T22:20:12.913 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:42 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:13.007 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rbd-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:13.060 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:13.065 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:13.065 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:13.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-fuse 2023-12-02T22:20:13.212 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:44 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:13.362 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 3.0 M 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:20:13.363 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:13.364 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.0 M 2023-12-02T22:20:13.364 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 12 M 2023-12-02T22:20:13.364 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:20:13.723 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:43 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:13.749 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-18.0.0-7670.g0c523c70.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2023-12-02T22:20:13.749 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:13.750 INFO:teuthology.orchestra.run.smithi170.stdout:Total 7.8 MB/s | 3.0 MB 00:00 2023-12-02T22:20:13.750 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:20:13.759 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:20:13.759 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:20:13.831 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:20:13.831 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:20:13.873 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 85 k 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:20:13.874 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:13.875 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:20:13.875 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:13.876 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 85 k 2023-12-02T22:20:13.876 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 231 k 2023-12-02T22:20:13.876 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:20:14.199 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-fuse-18.0.0-7670.g0c523c70.el9.x86_64.rpm 263 kB/s | 85 kB 00:00 2023-12-02T22:20:14.200 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:14.200 INFO:teuthology.orchestra.run.smithi012.stdout:Total 262 kB/s | 85 kB 00:00 2023-12-02T22:20:14.200 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:20:14.207 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:20:14.207 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:20:14.242 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:20:14.266 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:20:14.266 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:20:14.308 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:14.332 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:14.332 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:20:14.332 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-02T22:20:14.332 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T22:20:14.333 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T22:20:14.333 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:14.412 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:20:14.517 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:14.895 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:15.221 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:15.222 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:15.222 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:20:15.222 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:15.222 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:15.222 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:15.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-mirror 2023-12-02T22:20:15.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:15.442 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:15.442 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:20:15.442 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:15.443 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:15.443 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:15.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2023-12-02T22:20:16.033 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:46 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:16.185 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 3.0 M 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:20:16.186 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:16.187 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 3.0 M 2023-12-02T22:20:16.187 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 12 M 2023-12-02T22:20:16.187 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:20:16.294 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:47 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:16.445 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:16.446 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-02T22:20:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:16.446 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-02T22:20:16.446 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 170 k 2023-12-02T22:20:16.446 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 170 k 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 490 k 2023-12-02T22:20:16.447 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-02T22:20:16.678 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-18.0.0-7670.g0c523c70.el9.x86_64.rpm 737 kB/s | 170 kB 00:00 2023-12-02T22:20:16.679 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:16.679 INFO:teuthology.orchestra.run.smithi170.stdout:Total 730 kB/s | 170 kB 00:00 2023-12-02T22:20:16.679 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-02T22:20:16.684 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-mirror-18.0.0-7670.g0c523c70.el9.x86_64.rpm 6.0 MB/s | 3.0 MB 00:00 2023-12-02T22:20:16.685 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:16.685 INFO:teuthology.orchestra.run.smithi012.stdout:Total 6.0 MB/s | 3.0 MB 00:00 2023-12-02T22:20:16.686 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:20:16.690 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-02T22:20:16.690 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-02T22:20:16.696 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:20:16.696 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:20:16.746 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-02T22:20:16.746 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-02T22:20:16.773 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:20:16.773 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:20:16.896 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-02T22:20:17.002 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:17.209 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:20:17.287 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:17.310 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:17.311 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T22:20:17.311 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-02T22:20:17.311 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T22:20:17.311 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T22:20:17.311 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:17.423 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:17.701 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:17.701 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:17.701 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-02T22:20:17.701 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:17.701 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:17.701 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:17.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2023-12-02T22:20:18.498 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:18.499 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:18.499 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:20:18.499 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:18.499 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:18.499 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:18.569 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:49 ago on Sat 02 Dec 2023 10:18:29 PM UTC. 2023-12-02T22:20:18.667 INFO:teuthology.orchestra.run.smithi170.stdout:Package rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:18.720 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-02T22:20:18.720 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-02T22:20:18.720 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-02T22:20:18.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-nbd 2023-12-02T22:20:18.799 DEBUG:teuthology.parallel:result is None 2023-12-02T22:20:19.331 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:49 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:19.483 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:19.484 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2023-12-02T22:20:19.484 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:19.484 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-02T22:20:19.484 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:18.0.0-7670.g0c523c70.el9 ceph 170 k 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 170 k 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 490 k 2023-12-02T22:20:19.485 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-02T22:20:19.846 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-nbd-18.0.0-7670.g0c523c70.el9.x86_64.rpm 472 kB/s | 170 kB 00:00 2023-12-02T22:20:19.846 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-02T22:20:19.847 INFO:teuthology.orchestra.run.smithi012.stdout:Total 470 kB/s | 170 kB 00:00 2023-12-02T22:20:19.847 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-02T22:20:19.855 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-02T22:20:19.855 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-02T22:20:19.916 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-02T22:20:19.916 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-02T22:20:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-02T22:20:20.164 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 1/1 2023-12-02T22:20:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:20.986 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-02T22:20:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:18.0.0-7670.g0c523c70.el9.x86_64 2023-12-02T22:20:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:20.986 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:21.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-mirror 2023-12-02T22:20:21.798 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:51 ago on Sat 02 Dec 2023 10:18:30 PM UTC. 2023-12-02T22:20:21.898 INFO:teuthology.orchestra.run.smithi012.stdout:Package rbd-mirror-2:18.0.0-7670.g0c523c70.el9.x86_64 is already installed. 2023-12-02T22:20:21.953 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-02T22:20:21.953 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-12-02T22:20:21.954 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-02T22:20:22.016 DEBUG:teuthology.parallel:result is None 2023-12-02T22:20:22.017 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:20:22.167 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-02T22:20:22.189 INFO:teuthology.orchestra.run.smithi012.stdout:18.0.0-7670.g0c523c70.el9 2023-12-02T22:20:22.190 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7670.g0c523c70.el9 2023-12-02T22:20:22.190 INFO:teuthology.task.install:The correct ceph version 18.0.0-7670.g0c523c70 is installed. 2023-12-02T22:20:22.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:20:22.469 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-02T22:20:22.494 INFO:teuthology.orchestra.run.smithi170.stdout:18.0.0-7670.g0c523c70.el9 2023-12-02T22:20:22.494 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7670.g0c523c70.el9 2023-12-02T22:20:22.494 INFO:teuthology.task.install:The correct ceph version 18.0.0-7670.g0c523c70 is installed. 2023-12-02T22:20:22.494 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-02T22:20:22.494 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:22.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-02T22:20:22.526 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:22.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-02T22:20:22.566 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-02T22:20:22.566 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:22.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2023-12-02T22:20:22.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-02T22:20:22.672 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:22.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-12-02T22:20:22.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-02T22:20:22.777 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-02T22:20:22.778 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:22.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-02T22:20:22.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-02T22:20:22.881 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:22.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-02T22:20:22.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-02T22:20:22.981 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-02T22:20:22.981 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:22.981 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2023-12-02T22:20:23.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-02T22:20:23.085 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:23.085 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-12-02T22:20:23.115 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-02T22:20:23.185 INFO:teuthology.run_tasks:Running task ceph... 2023-12-02T22:20:23.285 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-02T22:20:23.286 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2023-12-02T22:20:23.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2023-12-02T22:20:23.321 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-02T22:20:23.321 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-02T22:20:23.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-02T22:20:23.391 INFO:tasks.ceph:Creating extra log directories... 2023-12-02T22:20:23.392 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-02T22:20:23.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-02T22:20:23.483 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-02T22:20:23.484 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-02T22:20:23.484 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7475805', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2667, 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95', 'sleep_before_teardown': 0, 'subset': '17/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXvhR6JCsTwPdNXSIOZvx45StfGsMUDwgQgZDSRvgDFGOjSjKQY32w4gnwhnr3iQ8P2nxx3y/yZOma8AInV3tM=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAO9CBiefD2/lvxCVlSKINHzBVGThnPqas2AV49ObW4POGhsYMBoxblDlHCPGauDDzzxu0VRdL2bZtXX0ZVQWQ0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-02_21:54:05', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-02T22:20:23.484 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-02T22:20:23.506 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-02T22:20:23.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T22:20:23.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T22:20:23.594 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:23.594 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2023-12-02T22:20:23.649 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-02T22:20:23.650 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2023-12-02T22:20:23.704 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-02T22:20:23.704 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:20:23.704 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-02T22:20:23.704 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:20:23.704 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:20:23.705 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-02 22:20:17.544827218 +0000 2023-12-02T22:20:23.705 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-02 22:16:53.625812465 +0000 2023-12-02T22:20:23.705 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-02 22:16:53.625812465 +0000 2023-12-02T22:20:23.705 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-02 22:16:53.625812465 +0000 2023-12-02T22:20:23.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-02T22:20:23.773 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-02T22:20:23.773 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-02T22:20:23.773 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000219221 s, 2.3 MB/s 2023-12-02T22:20:23.775 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-02T22:20:23.831 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2023-12-02T22:20:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-02T22:20:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:20:23.888 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-02T22:20:23.888 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:20:23.888 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:20:23.888 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-02 22:20:17.544827218 +0000 2023-12-02T22:20:23.889 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-02 22:16:55.068799458 +0000 2023-12-02T22:20:23.889 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-02 22:16:55.068799458 +0000 2023-12-02T22:20:23.889 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-02 22:16:55.068799458 +0000 2023-12-02T22:20:23.889 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-02T22:20:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-02T22:20:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-02T22:20:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000213329 s, 2.4 MB/s 2023-12-02T22:20:23.959 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-02T22:20:24.015 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2023-12-02T22:20:24.070 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-02T22:20:24.070 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:20:24.070 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-02T22:20:24.070 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:20:24.070 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:20:24.071 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-02 22:20:17.545827209 +0000 2023-12-02T22:20:24.071 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-02 22:16:56.550786099 +0000 2023-12-02T22:20:24.071 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-02 22:16:56.550786099 +0000 2023-12-02T22:20:24.071 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-02 22:16:56.550786099 +0000 2023-12-02T22:20:24.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-02T22:20:24.140 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-02T22:20:24.140 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-02T22:20:24.140 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000197807 s, 2.6 MB/s 2023-12-02T22:20:24.142 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-02T22:20:24.198 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2023-12-02T22:20:24.253 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-02T22:20:24.253 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-02 22:20:17.545827209 +0000 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-02 22:16:58.196771262 +0000 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-02 22:16:58.196771262 +0000 2023-12-02T22:20:24.254 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-02 22:16:58.196771262 +0000 2023-12-02T22:20:24.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-02T22:20:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-02T22:20:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-02T22:20:24.322 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000188144 s, 2.7 MB/s 2023-12-02T22:20:24.323 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-02T22:20:24.379 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:20:24.379 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T22:20:24.379 INFO:tasks.ceph:Generating config... 2023-12-02T22:20:24.380 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client] log to stderr = False 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.381 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.381 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-02T22:20:24.382 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-02T22:20:24.382 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-02T22:20:24.382 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-02T22:20:24.382 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-02T22:20:24.382 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-02T22:20:24.382 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-02T22:20:24.382 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-02T22:20:24.382 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-02T22:20:24.383 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-02T22:20:24.383 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-02T22:20:24.383 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-02T22:20:24.475 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/cluster1.keyring 2023-12-02T22:20:24.475 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-02T22:20:24.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-02T22:20:24.555 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.12')] 2023-12-02T22:20:24.555 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-02T22:20:24.556 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:24.556 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T22:20:24.610 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid 1b87a8aa-c41b-4602-a35b-641c227d62c2 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = pacific 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 1b87a8aa-c41b-4602-a35b-641c227d62c2 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2023-12-02T22:20:24.688726+0000 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:created 2023-12-02T22:20:24.688726+0000 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 16 (pacific) 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-02T22:20:24.694 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-02T22:20:24.695 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T22:20:24.749 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 1b87a8aa-c41b-4602-a35b-641c227d62c2... 2023-12-02T22:20:24.750 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-02T22:20:24.791 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-02T22:20:24.849 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-12-02T22:20:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-12-02T22:20:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T22:20:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-12-02T22:20:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T22:20:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.12 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on pool no app = False 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 5000 2023-12-02T22:20:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = 1b87a8aa-c41b-4602-a35b-641c227d62c2 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-12-02T22:20:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 1/20 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 1/20 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-12-02T22:20:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 4/10 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: osd memory target = 1610612736 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.856 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-12-02T22:20:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:20:24.859 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd = 20 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd_mirror = 15 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: log to stderr = False 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: rbd default features = 125 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: rbd mirror image policy type = simple 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2023-12-02T22:20:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.861 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2023-12-02T22:20:24.862 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2023-12-02T22:20:24.863 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.864 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = 172.21.15.12 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: mon client directed command retry = 5 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on pool no app = False 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject socket failures = 5000 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 1b87a8aa-c41b-4602-a35b-641c227d62c2 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2023-12-02T22:20:24.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2023-12-02T22:20:24.866 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.867 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: bdev async discard = True 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: bdev enable discard = True 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 1/20 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 1/20 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-02T22:20:24.868 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 4/10 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: osd memory target = 1610612736 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2023-12-02T22:20:24.869 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-12-02T22:20:24.870 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.871 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2023-12-02T22:20:24.872 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2023-12-02T22:20:24.873 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: debug rbd = 20 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: debug rbd_mirror = 15 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: log to stderr = False 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: rbd default features = 125 2023-12-02T22:20:24.874 INFO:teuthology.orchestra.run.smithi012.stdout: rbd mirror image policy type = simple 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.0] 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.1] 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.2] 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.3] 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:20:24.875 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:20:24.877 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-02T22:20:24.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-02T22:20:24.936 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-02T22:20:24.937 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:24.937 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-02T22:20:24.953 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:24.953 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-02T22:20:25.008 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T22:20:25.009 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:25.009 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-02T22:20:25.009 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-02T22:20:25.096 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:25.096 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:25.154 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:20:25.154 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:25.154 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-02T22:20:25.154 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-02T22:20:25.197 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:25.197 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:25.254 INFO:tasks.ceph:Setting up mon nodes... 2023-12-02T22:20:25.254 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-02T22:20:25.255 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-02T22:20:25.317 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-02T22:20:25.319 INFO:tasks.ceph:Setting up mds nodes... 2023-12-02T22:20:25.319 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-02T22:20:25.320 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-02T22:20:25.367 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-02T22:20:25.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-02T22:20:25.442 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-02T22:20:25.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-02T22:20:25.501 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-02T22:20:25.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-02T22:20:25.600 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-02T22:20:25.616 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-02T22:20:25.698 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-02T22:20:25.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-02T22:20:25.795 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-02T22:20:25.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-02T22:20:25.892 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-02T22:20:25.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-02T22:20:25.990 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-02T22:20:26.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-02T22:20:26.087 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-02T22:20:26.102 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-02T22:20:26.103 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T22:20:26.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-02T22:20:26.138 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:20:26.138 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-02T22:20:26.138 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T22:20:26.138 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-02T22:20:26.219 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T22:20:26.220 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T22:20:26.220 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T22:20:26.220 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T22:20:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-02T22:20:26.741 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-02T22:20:26.742 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-02T22:20:26.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-02T22:20:26.940 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-02T22:20:27.015 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:20:27.015 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-02T22:20:27.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T22:20:27.015 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-02T22:20:27.097 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T22:20:27.098 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T22:20:27.548 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-02T22:20:27.556 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-02T22:20:27.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-02T22:20:27.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-02T22:20:27.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-02T22:20:27.768 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:20:27.768 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-02T22:20:27.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T22:20:27.768 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-02T22:20:27.850 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T22:20:27.850 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T22:20:27.850 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T22:20:27.850 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T22:20:27.851 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T22:20:27.851 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-02T22:20:27.851 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T22:20:27.851 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T22:20:27.851 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T22:20:27.851 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T22:20:28.347 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-02T22:20:28.349 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-02T22:20:28.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-02T22:20:28.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-02T22:20:28.555 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:28.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:28.661+0000 7f5f7ed39500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-02T22:20:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:28.665+0000 7f5f7ed39500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-02T22:20:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:28.665+0000 7f5f7ed39500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-02T22:20:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:28.665+0000 7f5f7ed39500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-02T22:20:28.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:28.665+0000 7f5f7ed39500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-02T22:20:28.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:28.673+0000 7f5f7ed39500 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-02T22:20:31.526 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-02T22:20:31.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:31.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:31.656+0000 7f7eb8cc5500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-02T22:20:31.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:31.657+0000 7f7eb8cc5500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-02T22:20:31.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:31.657+0000 7f7eb8cc5500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-02T22:20:31.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:31.657+0000 7f7eb8cc5500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-02T22:20:31.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:31.657+0000 7f7eb8cc5500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-02T22:20:31.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:31.657+0000 7f7eb8cc5500 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-02T22:20:34.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-02T22:20:34.564 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:34.658+0000 7f93b9a1f500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-02T22:20:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:34.658+0000 7f93b9a1f500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-02T22:20:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:34.658+0000 7f93b9a1f500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-02T22:20:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:34.658+0000 7f93b9a1f500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-02T22:20:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:34.658+0000 7f93b9a1f500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-02T22:20:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-02T22:20:34.659+0000 7f93b9a1f500 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-02T22:20:37.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-02T22:20:37.542 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-02T22:20:37.542 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:37.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-02T22:20:37.617 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:37.617 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-02T22:20:37.690 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:37.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-02T22:20:37.762 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:37.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-02T22:20:37.833 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:37.833 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-02T22:20:37.890 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:37.890 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-02T22:20:37.910 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:37.910 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-02T22:20:37.967 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:37.968 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-02T22:20:38.026 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:38.026 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-02T22:20:38.083 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:38.084 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-02T22:20:38.142 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:38.143 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-02T22:20:38.199 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:38.199 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-02T22:20:38.257 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:20:38.257 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-02T22:20:38.313 INFO:tasks.ceph:Adding keys to all mons... 2023-12-02T22:20:38.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-02T22:20:38.345 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlnYHiEhAAz4bORwbQPesmionk2nX9cA== 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCsrWtl33t+JxAAI6x9qd56+Jl2PpJEV5CoBA== 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCvrWtlHTM1JxAAvgBsjgns9oRPfqXrAOZK0w== 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCyrWtlgRZMJxAAv7mnhGhaXmDJlEo+Xt+CEQ== 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlGDPhFRAAvVQbvG8VHjeuV3cz18em3g== 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror] 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlkkc4GhAAkjQNNcwngaveBH3hHGq9Ow== 2023-12-02T22:20:38.346 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.0] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlSH7ZHRAAOMIGw73UjsDQ69Xnp+AwMg== 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.1] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlr03HIxAA6XNUrBnS4oFDS52XUdQCAg== 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.2] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtldNGXKRAArjrGzbcFao1CydNSNPOWyQ== 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.3] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlNptiLxAASflO3FjF+0v9lrZXnIVR6w== 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.4] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtlCsYrNRAAoZMpvG5/dC5Q9nN0wX/JCg== 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.5] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCprWtl0EMAOxAAw4bbi4QTnera2Bq5DPrzZA== 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.6] 2023-12-02T22:20:38.347 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCqrWtl2TAzBRAA0vyqac49Z7nRTX3rhgWRUw== 2023-12-02T22:20:38.348 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-02T22:20:38.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T22:20:38.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T22:20:38.566 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T22:20:38.622 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:38.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:38.739 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:38.799 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:38.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:38.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:38.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:39.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:39.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:20:39.158 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-02T22:20:39.158 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-02T22:20:39.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-02T22:20:39.293 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-02T22:20:39.323 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T22:20:39.379 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-02T22:20:39.380 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-02T22:20:39.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-02T22:20:39.422 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-02T22:20:39.422 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-02T22:20:39.422 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-02T22:20:39.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-02T22:20:39.426 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-02T22:20:39.426 INFO:tasks.ceph:Setting crush tunables to default 2023-12-02T22:20:39.427 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-02T22:20:39.623 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2023-12-02T22:20:39.634 INFO:tasks.ceph:check_enable_crimson: False 2023-12-02T22:20:39.634 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-02T22:20:39.634 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:39.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-02T22:20:39.669 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:39.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-02T22:20:39.741 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:20:39.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-02T22:20:39.812 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 osd new e04d4736-d721-4737-913f-88be3aed59dc 0 2023-12-02T22:20:40.040 INFO:teuthology.orchestra.run.smithi012.stdout:0 2023-12-02T22:20:40.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 osd new 816087c7-a78c-48b7-9ad4-b24ff37a888a 1 2023-12-02T22:20:40.201 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:40.195+0000 7fe3d0c56040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-02T22:20:40.245 INFO:teuthology.orchestra.run.smithi012.stdout:1 2023-12-02T22:20:40.251 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:40.249+0000 7fe3d0c56040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-02T22:20:40.256 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 osd new e9bdc8c4-9464-4db4-98b0-82dd4154703d 2 2023-12-02T22:20:40.443 INFO:teuthology.orchestra.run.smithi012.stdout:2 2023-12-02T22:20:40.455 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-02T22:20:40.455 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-02T22:20:40.457 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-02T22:20:40.457 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-02T22:20:40.457 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-02T22:20:40.461 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-02T22:20:40.462 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-02T22:20:40.462 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-02T22:20:40.464 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-02T22:20:40.464 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T22:20:40.676 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:40.676 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:20:40.442626+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:20:40.691 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-02T22:20:40.692 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-02T22:20:40.863 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:40.861+0000 7fe3d0c56040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-02T22:20:40.917 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:40.916+0000 7fe3d0c56040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-02T22:20:40.964 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:40.962+0000 7fe3d0c56040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-02T22:20:40.997 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-02T22:20:41.020 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.018+0000 7fe3d0c56040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-02T22:20:41.063 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:20:41.061+0000 7f1ee4abe500 -1 Falling back to public interface 2023-12-02T22:20:41.064 INFO:tasks.ceph.cluster1.osd.1.smithi012.stderr:2023-12-02T22:20:41.063+0000 7f651580e500 -1 Falling back to public interface 2023-12-02T22:20:41.066 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.065+0000 7fe3d0c56040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-02T22:20:41.073 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:20:41.070+0000 7eff4beab500 -1 Falling back to public interface 2023-12-02T22:20:41.160 INFO:teuthology.misc.health.smithi012.stdout: 2023-12-02T22:20:41.160 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":5,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:20:40.442626+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:20:41.170 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T22:20:41.176 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.175+0000 7fe3d0c56040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-02T22:20:41.249 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.248+0000 7fe3d0c56040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-02T22:20:41.297 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.295+0000 7fe3d0c56040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-02T22:20:41.351 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.350+0000 7fe3d0c56040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-02T22:20:41.400 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.399+0000 7fe3d0c56040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-02T22:20:41.799 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.798+0000 7fe3d0c56040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-02T22:20:41.941 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:41.940+0000 7fe3d0c56040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-02T22:20:42.065 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.064+0000 7fe3d0c56040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-02T22:20:42.112 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.111+0000 7fe3d0c56040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-02T22:20:42.214 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.213+0000 7fe3d0c56040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-02T22:20:42.313 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.312+0000 7fe3d0c56040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-02T22:20:42.460 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.458+0000 7fe3d0c56040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-02T22:20:42.504 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.502+0000 7fe3d0c56040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-02T22:20:42.650 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:42.649+0000 7fe3d0c56040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-02T22:20:43.482 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:43.479+0000 7fe3d0c56040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-02T22:20:43.774 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:43.773+0000 7fe3d0c56040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-02T22:20:43.838 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:43.837+0000 7fe3d0c56040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-02T22:20:43.945 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:43.944+0000 7fe3d0c56040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-02T22:20:44.557 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:20:44.555+0000 7f1ee4abe500 -1 osd.0 0 log_to_monitors true 2023-12-02T22:20:44.559 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:20:44.558+0000 7eff4beab500 -1 osd.2 0 log_to_monitors true 2023-12-02T22:20:44.585 INFO:tasks.ceph.cluster1.osd.1.smithi012.stderr:2023-12-02T22:20:44.584+0000 7f651580e500 -1 osd.1 0 log_to_monitors true 2023-12-02T22:20:44.967 INFO:tasks.ceph.cluster1.mgr.x.smithi012.stderr:2023-12-02T22:20:44.963+0000 7fe3720bc640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-02T22:20:47.473 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-02T22:20:47.904 INFO:teuthology.misc.health.smithi012.stdout: 2023-12-02T22:20:47.904 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":7,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:20:46.514824+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:20:47.920 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T22:20:48.358 INFO:tasks.ceph.cluster1.osd.1.smithi012.stderr:2023-12-02T22:20:48.357+0000 7f6510986640 -1 osd.1 0 waiting for initial osdmap 2023-12-02T22:20:48.373 INFO:tasks.ceph.cluster1.osd.1.smithi012.stderr:2023-12-02T22:20:48.371+0000 7f6508fbf640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T22:20:48.438 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:20:48.437+0000 7f1ee0229640 -1 osd.0 0 waiting for initial osdmap 2023-12-02T22:20:48.455 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:20:48.454+0000 7f1ed8862640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T22:20:48.474 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:20:48.472+0000 7eff47616640 -1 osd.2 0 waiting for initial osdmap 2023-12-02T22:20:48.488 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:20:48.486+0000 7eff3fc4f640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T22:20:54.224 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-02T22:20:54.516 INFO:teuthology.misc.health.smithi012.stdout: 2023-12-02T22:20:54.516 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":11,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:20:52.536717+0000","last_up_change":"2023-12-02T22:20:49.358782+0000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:20:50.019027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6801","nonce":3782931675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6803","nonce":3782931675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6807","nonce":3782931675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6805","nonce":3782931675}]},"public_addr":"172.21.15.12:6801/3782931675","cluster_addr":"172.21.15.12:6803/3782931675","heartbeat_back_addr":"172.21.15.12:6807/3782931675","heartbeat_front_addr":"172.21.15.12:6805/3782931675","state":["exists","up"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6809","nonce":3639128561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6811","nonce":3639128561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6815","nonce":3639128561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6813","nonce":3639128561}]},"public_addr":"172.21.15.12:6809/3639128561","cluster_addr":"172.21.15.12:6811/3639128561","heartbeat_back_addr":"172.21.15.12:6815/3639128561","heartbeat_front_addr":"172.21.15.12:6813/3639128561","state":["exists","up"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6817","nonce":4018360220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6819","nonce":4018360220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6823","nonce":4018360220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6821","nonce":4018360220}]},"public_addr":"172.21.15.12:6817/4018360220","cluster_addr":"172.21.15.12:6819/4018360220","heartbeat_back_addr":"172.21.15.12:6823/4018360220","heartbeat_front_addr":"172.21.15.12:6821/4018360220","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.546021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.605275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.577859+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:20:54.527 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-02T22:20:54.527 INFO:tasks.ceph:Creating RBD pool 2023-12-02T22:20:54.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-02T22:20:55.547 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'rbd' created 2023-12-02T22:20:55.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-02T22:20:56.552 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-02T22:20:56.572 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-02T22:20:56.573 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-02T22:20:56.573 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-02T22:20:56.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-02T22:20:56.573 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-02T22:20:56.923 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:56.935 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":2387357575},{"type":"v1","addr":"172.21.15.12:6825","nonce":2387357575}]},"active_addr":"172.21.15.12:6825/2387357575","active_change":"2023-12-02T22:20:43.957646+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3376669875}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3357566268}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":708161763}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2898495949}]}]} 2023-12-02T22:20:56.938 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-02T22:20:56.938 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-02T22:20:56.938 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T22:20:57.234 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:57.235 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:20:56.546529+0000","last_up_change":"2023-12-02T22:20:49.358782+0000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:20:50.019027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T22:20:54.851652+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6801","nonce":3782931675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6803","nonce":3782931675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6807","nonce":3782931675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6805","nonce":3782931675}]},"public_addr":"172.21.15.12:6801/3782931675","cluster_addr":"172.21.15.12:6803/3782931675","heartbeat_back_addr":"172.21.15.12:6807/3782931675","heartbeat_front_addr":"172.21.15.12:6805/3782931675","state":["exists","up"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6809","nonce":3639128561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6811","nonce":3639128561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6815","nonce":3639128561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6813","nonce":3639128561}]},"public_addr":"172.21.15.12:6809/3639128561","cluster_addr":"172.21.15.12:6811/3639128561","heartbeat_back_addr":"172.21.15.12:6815/3639128561","heartbeat_front_addr":"172.21.15.12:6813/3639128561","state":["exists","up"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6817","nonce":4018360220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6819","nonce":4018360220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6823","nonce":4018360220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6821","nonce":4018360220}]},"public_addr":"172.21.15.12:6817/4018360220","cluster_addr":"172.21.15.12:6819/4018360220","heartbeat_back_addr":"172.21.15.12:6823/4018360220","heartbeat_front_addr":"172.21.15.12:6821/4018360220","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.546021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.605275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.577859+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:20:57.248 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-02T22:20:57.248 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T22:20:57.547 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:57.547 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:20:56.546529+0000","last_up_change":"2023-12-02T22:20:49.358782+0000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:20:50.019027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T22:20:54.851652+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6801","nonce":3782931675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6803","nonce":3782931675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6807","nonce":3782931675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6805","nonce":3782931675}]},"public_addr":"172.21.15.12:6801/3782931675","cluster_addr":"172.21.15.12:6803/3782931675","heartbeat_back_addr":"172.21.15.12:6807/3782931675","heartbeat_front_addr":"172.21.15.12:6805/3782931675","state":["exists","up"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6809","nonce":3639128561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6811","nonce":3639128561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6815","nonce":3639128561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6813","nonce":3639128561}]},"public_addr":"172.21.15.12:6809/3639128561","cluster_addr":"172.21.15.12:6811/3639128561","heartbeat_back_addr":"172.21.15.12:6815/3639128561","heartbeat_front_addr":"172.21.15.12:6813/3639128561","state":["exists","up"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6817","nonce":4018360220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6819","nonce":4018360220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6823","nonce":4018360220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6821","nonce":4018360220}]},"public_addr":"172.21.15.12:6817/4018360220","cluster_addr":"172.21.15.12:6819/4018360220","heartbeat_back_addr":"172.21.15.12:6823/4018360220","heartbeat_front_addr":"172.21.15.12:6821/4018360220","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.546021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.605275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.577859+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:20:57.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-02T22:20:57.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-02T22:20:57.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-02T22:20:57.709 INFO:teuthology.orchestra.run.smithi012.stdout:34359738373 2023-12-02T22:20:57.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-02T22:20:57.712 INFO:teuthology.orchestra.run.smithi012.stdout:34359738373 2023-12-02T22:20:57.713 INFO:teuthology.orchestra.run.smithi012.stdout:34359738374 2023-12-02T22:20:57.713 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-02T22:20:57.714 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-02T22:20:58.018 INFO:teuthology.orchestra.run.smithi012.stdout:34359738372 2023-12-02T22:20:58.031 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2023-12-02T22:20:58.058 INFO:teuthology.orchestra.run.smithi012.stdout:34359738373 2023-12-02T22:20:58.060 INFO:teuthology.orchestra.run.smithi012.stdout:34359738372 2023-12-02T22:20:58.072 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.1 2023-12-02T22:20:58.073 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2023-12-02T22:20:59.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-02T22:20:59.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-02T22:20:59.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-02T22:20:59.352 INFO:teuthology.orchestra.run.smithi012.stdout:34359738373 2023-12-02T22:20:59.366 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2023-12-02T22:20:59.366 DEBUG:teuthology.parallel:result is None 2023-12-02T22:20:59.406 INFO:teuthology.orchestra.run.smithi012.stdout:34359738373 2023-12-02T22:20:59.410 INFO:teuthology.orchestra.run.smithi012.stdout:34359738374 2023-12-02T22:20:59.420 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2023-12-02T22:20:59.420 DEBUG:teuthology.parallel:result is None 2023-12-02T22:20:59.421 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2023-12-02T22:20:59.421 DEBUG:teuthology.parallel:result is None 2023-12-02T22:20:59.421 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-02T22:20:59.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T22:20:59.752 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:20:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T22:20:59.764 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-02T22:20:57.963767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82572,"kb_used_data":1504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032948,"statfs":{"total":289910292480,"available":289825738752,"internally_reserved":0,"allocated":1540096,"data_stored":1278207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.414818"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554980+0000","last_change":"2023-12-02T22:20:56.564305+0000","last_active":"2023-12-02T22:20:57.554980+0000","last_peered":"2023-12-02T22:20:57.554980+0000","last_clean":"2023-12-02T22:20:57.554980+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T22:20:57.554980+0000","last_undegraded":"2023-12-02T22:20:57.554980+0000","last_fullsized":"2023-12-02T22:20:57.554980+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:31:54.028189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554957+0000","last_change":"2023-12-02T22:20:56.565011+0000","last_active":"2023-12-02T22:20:57.554957+0000","last_peered":"2023-12-02T22:20:57.554957+0000","last_clean":"2023-12-02T22:20:57.554957+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T22:20:57.554957+0000","last_undegraded":"2023-12-02T22:20:57.554957+0000","last_fullsized":"2023-12-02T22:20:57.554957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:00:19.409954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.555030+0000","last_change":"2023-12-02T22:20:56.563555+0000","last_active":"2023-12-02T22:20:57.555030+0000","last_peered":"2023-12-02T22:20:57.555030+0000","last_clean":"2023-12-02T22:20:57.555030+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T22:20:57.555030+0000","last_undegraded":"2023-12-02T22:20:57.555030+0000","last_fullsized":"2023-12-02T22:20:57.555030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T09:35:24.530609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554982+0000","last_change":"2023-12-02T22:20:56.562760+0000","last_active":"2023-12-02T22:20:57.554982+0000","last_peered":"2023-12-02T22:20:57.554982+0000","last_clean":"2023-12-02T22:20:57.554982+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T22:20:57.554982+0000","last_undegraded":"2023-12-02T22:20:57.554982+0000","last_fullsized":"2023-12-02T22:20:57.554982+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:42:42.364951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:20:56.563870+0000","last_change":"2023-12-02T22:20:56.562058+0000","last_active":"2023-12-02T22:20:56.563870+0000","last_peered":"2023-12-02T22:20:56.563870+0000","last_clean":"2023-12-02T22:20:56.563870+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T22:20:56.563870+0000","last_undegraded":"2023-12-02T22:20:56.563870+0000","last_fullsized":"2023-12-02T22:20:56.563870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:37:18.800980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:20:56.566059+0000","last_change":"2023-12-02T22:20:56.563338+0000","last_active":"2023-12-02T22:20:56.566059+0000","last_peered":"2023-12-02T22:20:56.566059+0000","last_clean":"2023-12-02T22:20:56.566059+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T22:20:56.566059+0000","last_undegraded":"2023-12-02T22:20:56.566059+0000","last_fullsized":"2023-12-02T22:20:56.566059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:32.220019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:20:56.566479+0000","last_change":"2023-12-02T22:20:56.565388+0000","last_active":"2023-12-02T22:20:56.566479+0000","last_peered":"2023-12-02T22:20:56.566479+0000","last_clean":"2023-12-02T22:20:56.566479+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T22:20:56.566479+0000","last_undegraded":"2023-12-02T22:20:56.566479+0000","last_fullsized":"2023-12-02T22:20:56.566479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:05:26.496389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554912+0000","last_change":"2023-12-02T22:20:56.563039+0000","last_active":"2023-12-02T22:20:57.554912+0000","last_peered":"2023-12-02T22:20:57.554912+0000","last_clean":"2023-12-02T22:20:57.554912+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T22:20:57.554912+0000","last_undegraded":"2023-12-02T22:20:57.554912+0000","last_fullsized":"2023-12-02T22:20:57.554912+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:37:53.638783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:20:56.560678+0000","last_change":"2023-12-02T22:20:51.698937+0000","last_active":"2023-12-02T22:20:56.560678+0000","last_peered":"2023-12-02T22:20:56.560678+0000","last_clean":"2023-12-02T22:20:56.560678+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T22:20:56.560678+0000","last_undegraded":"2023-12-02T22:20:56.560678+0000","last_fullsized":"2023-12-02T22:20:56.560678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_clean_scrub_stamp":"2023-12-02T22:20:50.530145+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:30:49.984827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:20:59.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T22:21:00.057 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:00.057 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T22:21:00.069 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-02T22:20:59.964141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82588,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032932,"statfs":{"total":289910292480,"available":289825722368,"internally_reserved":0,"allocated":1556480,"data_stored":1285163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.415193"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554980+0000","last_change":"2023-12-02T22:20:56.564305+0000","last_active":"2023-12-02T22:20:57.554980+0000","last_peered":"2023-12-02T22:20:57.554980+0000","last_clean":"2023-12-02T22:20:57.554980+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T22:20:57.554980+0000","last_undegraded":"2023-12-02T22:20:57.554980+0000","last_fullsized":"2023-12-02T22:20:57.554980+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:31:54.028189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554957+0000","last_change":"2023-12-02T22:20:56.565011+0000","last_active":"2023-12-02T22:20:57.554957+0000","last_peered":"2023-12-02T22:20:57.554957+0000","last_clean":"2023-12-02T22:20:57.554957+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T22:20:57.554957+0000","last_undegraded":"2023-12-02T22:20:57.554957+0000","last_fullsized":"2023-12-02T22:20:57.554957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:00:19.409954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.555030+0000","last_change":"2023-12-02T22:20:56.563555+0000","last_active":"2023-12-02T22:20:57.555030+0000","last_peered":"2023-12-02T22:20:57.555030+0000","last_clean":"2023-12-02T22:20:57.555030+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T22:20:57.555030+0000","last_undegraded":"2023-12-02T22:20:57.555030+0000","last_fullsized":"2023-12-02T22:20:57.555030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T09:35:24.530609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554982+0000","last_change":"2023-12-02T22:20:56.562760+0000","last_active":"2023-12-02T22:20:57.554982+0000","last_peered":"2023-12-02T22:20:57.554982+0000","last_clean":"2023-12-02T22:20:57.554982+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T22:20:57.554982+0000","last_undegraded":"2023-12-02T22:20:57.554982+0000","last_fullsized":"2023-12-02T22:20:57.554982+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:42:42.364951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.789369+0000","last_change":"2023-12-02T22:20:56.562058+0000","last_active":"2023-12-02T22:20:57.789369+0000","last_peered":"2023-12-02T22:20:57.789369+0000","last_clean":"2023-12-02T22:20:57.789369+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T22:20:57.789369+0000","last_undegraded":"2023-12-02T22:20:57.789369+0000","last_fullsized":"2023-12-02T22:20:57.789369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:09:32.035065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.789319+0000","last_change":"2023-12-02T22:20:56.563338+0000","last_active":"2023-12-02T22:20:57.789319+0000","last_peered":"2023-12-02T22:20:57.789319+0000","last_clean":"2023-12-02T22:20:57.789319+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T22:20:57.789319+0000","last_undegraded":"2023-12-02T22:20:57.789319+0000","last_fullsized":"2023-12-02T22:20:57.789319+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T23:03:15.596248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.789214+0000","last_change":"2023-12-02T22:20:56.565388+0000","last_active":"2023-12-02T22:20:57.789214+0000","last_peered":"2023-12-02T22:20:57.789214+0000","last_clean":"2023-12-02T22:20:57.789214+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T22:20:57.789214+0000","last_undegraded":"2023-12-02T22:20:57.789214+0000","last_fullsized":"2023-12-02T22:20:57.789214+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:24:24.875735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:20:57.554912+0000","last_change":"2023-12-02T22:20:56.563039+0000","last_active":"2023-12-02T22:20:57.554912+0000","last_peered":"2023-12-02T22:20:57.554912+0000","last_clean":"2023-12-02T22:20:57.554912+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T22:20:57.554912+0000","last_undegraded":"2023-12-02T22:20:57.554912+0000","last_fullsized":"2023-12-02T22:20:57.554912+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:37:53.638783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:20:56.560678+0000","last_change":"2023-12-02T22:20:51.698937+0000","last_active":"2023-12-02T22:20:56.560678+0000","last_peered":"2023-12-02T22:20:56.560678+0000","last_clean":"2023-12-02T22:20:56.560678+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T22:20:56.560678+0000","last_undegraded":"2023-12-02T22:20:56.560678+0000","last_fullsized":"2023-12-02T22:20:56.560678+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_clean_scrub_stamp":"2023-12-02T22:20:50.530145+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:30:49.984827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":34809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:21:00.070 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-02T22:21:00.070 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-02T22:21:00.070 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-02T22:21:00.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-02T22:21:00.398 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:00.398 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-02T22:21:00.411 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-02T22:21:00.411 INFO:teuthology.run_tasks:Running task ceph... 2023-12-02T22:21:00.422 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-02T22:21:00.422 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-02T22:21:00.423 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7475805', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2667, 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95', 'sleep_before_teardown': 0, 'subset': '17/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXvhR6JCsTwPdNXSIOZvx45StfGsMUDwgQgZDSRvgDFGOjSjKQY32w4gnwhnr3iQ8P2nxx3y/yZOma8AInV3tM=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAO9CBiefD2/lvxCVlSKINHzBVGThnPqas2AV49ObW4POGhsYMBoxblDlHCPGauDDzzxu0VRdL2bZtXX0ZVQWQ0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-02_21:54:05', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-02T22:21:00.424 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-02T22:21:00.427 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-02T22:21:00.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T22:21:00.482 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T22:21:00.517 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:00.517 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2023-12-02T22:21:00.575 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-02T22:21:00.575 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-02 22:20:14.563975880 +0000 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-02 22:16:38.412271198 +0000 2023-12-02T22:21:00.631 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-02 22:16:38.412271198 +0000 2023-12-02T22:21:00.632 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-12-02 22:16:38.412271198 +0000 2023-12-02T22:21:00.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-02T22:21:00.700 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-02T22:21:00.701 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-02T22:21:00.701 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.8723e-05 s, 5.8 MB/s 2023-12-02T22:21:00.702 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-02T22:21:00.759 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2023-12-02T22:21:00.815 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-02T22:21:00.815 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:21:00.815 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-02T22:21:00.815 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:21:00.815 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:21:00.815 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-02 22:20:14.563975880 +0000 2023-12-02T22:21:00.816 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-02 22:16:39.867237363 +0000 2023-12-02T22:21:00.816 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-02 22:16:39.867237363 +0000 2023-12-02T22:21:00.816 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-12-02 22:16:39.867237363 +0000 2023-12-02T22:21:00.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-02T22:21:00.883 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-02T22:21:00.883 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-02T22:21:00.883 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.9754e-05 s, 5.1 MB/s 2023-12-02T22:21:00.884 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-02T22:21:00.940 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2023-12-02T22:21:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-02 22:20:14.564975856 +0000 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-02 22:16:41.496199304 +0000 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-02 22:16:41.496199304 +0000 2023-12-02T22:21:00.996 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-12-02 22:16:41.496199304 +0000 2023-12-02T22:21:00.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-02T22:21:01.063 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-02T22:21:01.063 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-02T22:21:01.063 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000105106 s, 4.9 MB/s 2023-12-02T22:21:01.063 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-02T22:21:01.119 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-02 22:20:14.564975856 +0000 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-02 22:16:43.161160404 +0000 2023-12-02T22:21:01.174 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-02 22:16:43.161160404 +0000 2023-12-02T22:21:01.175 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-12-02 22:16:43.161160404 +0000 2023-12-02T22:21:01.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-02T22:21:01.239 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-02T22:21:01.239 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-02T22:21:01.239 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.2316e-05 s, 6.2 MB/s 2023-12-02T22:21:01.240 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-02T22:21:01.297 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:21:01.297 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T22:21:01.297 INFO:tasks.ceph:Generating config... 2023-12-02T22:21:01.298 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-02T22:21:01.298 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client] log to stderr = False 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.299 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.299 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-02T22:21:01.299 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-02T22:21:01.299 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-02T22:21:01.299 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-02T22:21:01.300 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-02T22:21:01.300 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-02T22:21:01.300 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-02T22:21:01.300 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-02T22:21:01.300 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-02T22:21:01.301 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-02T22:21:01.301 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-02T22:21:01.301 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-02T22:21:01.301 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-02T22:21:01.301 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-02T22:21:01.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-02T22:21:01.381 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.keyring 2023-12-02T22:21:01.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-02T22:21:01.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-02T22:21:01.538 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.170')] 2023-12-02T22:21:01.538 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.170', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-02T22:21:01.539 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:01.539 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T22:21:01.593 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.170 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: generated fsid 5e1eb4f6-095d-4fc4-8d62-43f3f257587e 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:setting min_mon_release = pacific 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 0 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:fsid 5e1eb4f6-095d-4fc4-8d62-43f3f257587e 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:last_changed 2023-12-02T22:21:01.671366+0000 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:created 2023-12-02T22:21:01.671366+0000 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:min_mon_release 16 (pacific) 2023-12-02T22:21:01.672 INFO:teuthology.orchestra.run.smithi170.stdout:election_strategy: 1 2023-12-02T22:21:01.673 INFO:teuthology.orchestra.run.smithi170.stdout:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-12-02T22:21:01.673 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-02T22:21:01.674 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T22:21:01.728 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 5e1eb4f6-095d-4fc4-8d62-43f3f257587e... 2023-12-02T22:21:01.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-02T22:21:01.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-02T22:21:01.799 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2023-12-02T22:21:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2023-12-02T22:21:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T22:21:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2023-12-02T22:21:01.801 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.802 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = 172.21.15.170 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: mon client directed command retry = 5 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on pool no app = False 2023-12-02T22:21:01.803 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject socket failures = 5000 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 5e1eb4f6-095d-4fc4-8d62-43f3f257587e 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.804 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2023-12-02T22:21:01.805 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2023-12-02T22:21:01.805 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.805 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.806 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2023-12-02T22:21:01.807 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.807 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2023-12-02T22:21:01.807 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.807 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2023-12-02T22:21:01.807 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2023-12-02T22:21:01.808 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: bdev async discard = True 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: bdev enable discard = True 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 1/20 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 1/20 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 4/10 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2023-12-02T22:21:01.809 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: osd memory target = 1610612736 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2023-12-02T22:21:01.810 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T22:21:01.811 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T22:21:01.812 INFO:teuthology.orchestra.run.smithi012.stdout: debug rbd = 20 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: debug rbd_mirror = 15 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: log to stderr = False 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: rbd default features = 125 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: rbd mirror image policy type = simple 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.0] 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.1] 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.2] 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.813 INFO:teuthology.orchestra.run.smithi012.stdout:[client.mirror.3] 2023-12-02T22:21:01.814 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.814 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.819 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-12-02T22:21:01.823 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-12-02T22:21:01.823 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T22:21:01.823 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-12-02T22:21:01.823 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.823 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-12-02T22:21:01.823 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T22:21:01.824 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.825 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-12-02T22:21:01.826 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.170 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on pool no app = False 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 5000 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = 5e1eb4f6-095d-4fc4-8d62-43f3f257587e 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-12-02T22:21:01.827 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2023-12-02T22:21:01.831 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.831 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-12-02T22:21:01.831 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-12-02T22:21:01.831 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.831 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-12-02T22:21:01.831 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.832 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-12-02T22:21:01.833 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-12-02T22:21:01.833 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.833 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-12-02T22:21:01.833 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-12-02T22:21:01.833 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 1/20 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 1/20 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 4/10 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T22:21:01.834 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: osd memory target = 1610612736 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.835 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T22:21:01.836 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-12-02T22:21:01.837 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd = 20 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd_mirror = 15 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: log to stderr = False 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: rbd default features = 125 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: rbd mirror image policy type = simple 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2023-12-02T22:21:01.838 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.839 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.839 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2023-12-02T22:21:01.839 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T22:21:01.839 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T22:21:01.840 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-02T22:21:01.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-02T22:21:01.898 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-02T22:21:01.898 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:01.898 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-02T22:21:01.913 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:01.913 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-02T22:21:01.968 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T22:21:01.969 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:21:01.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-02T22:21:01.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-02T22:21:02.019 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:21:02.019 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:02.076 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:21:02.076 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:02.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-02T22:21:02.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-02T22:21:02.116 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:02.117 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:02.173 INFO:tasks.ceph:Setting up mon nodes... 2023-12-02T22:21:02.173 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-02T22:21:02.173 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-02T22:21:02.237 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-02T22:21:02.240 INFO:tasks.ceph:Setting up mds nodes... 2023-12-02T22:21:02.240 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-02T22:21:02.240 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-02T22:21:02.287 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-02T22:21:02.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-02T22:21:02.350 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-02T22:21:02.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-02T22:21:02.449 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-02T22:21:02.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-02T22:21:02.546 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-02T22:21:02.562 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-02T22:21:02.644 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-02T22:21:02.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-02T22:21:02.741 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-02T22:21:02.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-02T22:21:02.838 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-02T22:21:02.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-02T22:21:02.938 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-02T22:21:02.953 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-02T22:21:03.036 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-02T22:21:03.050 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-02T22:21:03.051 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T22:21:03.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-02T22:21:03.118 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:21:03.118 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-02T22:21:03.118 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:21:03.118 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T22:21:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T22:21:03.186 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T22:21:03.217 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-12-02T22:21:03.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-12-02T22:21:03.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-02T22:21:03.294 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-02T22:21:03.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-02T22:21:03.436 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:21:03.436 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-02T22:21:03.436 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:21:03.437 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-02T22:21:03.504 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T22:21:03.505 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T22:21:03.537 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-12-02T22:21:03.539 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-12-02T22:21:03.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-02T22:21:03.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-02T22:21:03.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-02T22:21:03.750 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T22:21:03.750 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-02T22:21:03.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:21:03.751 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-02T22:21:03.818 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T22:21:03.819 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T22:21:03.850 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-12-02T22:21:03.852 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-12-02T22:21:03.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-02T22:21:03.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-02T22:21:03.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:04.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:04.089+0000 7f8519a25500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-02T22:21:04.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:04.089+0000 7f8519a25500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-02T22:21:04.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:04.089+0000 7f8519a25500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-02T22:21:04.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:04.089+0000 7f8519a25500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-02T22:21:04.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:04.089+0000 7f8519a25500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-02T22:21:04.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:04.089+0000 7f8519a25500 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-02T22:21:06.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-02T22:21:06.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:07.057+0000 7fcf30c14500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-02T22:21:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:07.058+0000 7fcf30c14500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-02T22:21:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:07.058+0000 7fcf30c14500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-02T22:21:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:07.058+0000 7fcf30c14500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-02T22:21:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:07.058+0000 7fcf30c14500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-02T22:21:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:07.058+0000 7fcf30c14500 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-02T22:21:09.903 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-02T22:21:09.934 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:10.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:10.023+0000 7fcd171be500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-02T22:21:10.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:10.023+0000 7fcd171be500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-02T22:21:10.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:10.023+0000 7fcd171be500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-02T22:21:10.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:10.023+0000 7fcd171be500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-02T22:21:10.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:10.023+0000 7fcd171be500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-02T22:21:10.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-02T22:21:10.024+0000 7fcd171be500 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-02T22:21:12.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-02T22:21:12.924 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-02T22:21:12.924 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:21:12.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-02T22:21:12.974 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:12.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-02T22:21:13.012 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-02T22:21:13.082 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.082 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-02T22:21:13.158 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-02T22:21:13.158 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-02T22:21:13.176 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.176 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-02T22:21:13.215 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.215 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-02T22:21:13.272 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.272 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-02T22:21:13.327 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.327 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-02T22:21:13.385 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.385 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-02T22:21:13.439 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.439 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-02T22:21:13.495 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.496 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-02T22:21:13.551 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:13.552 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-02T22:21:13.608 INFO:tasks.ceph:Adding keys to all mons... 2023-12-02T22:21:13.608 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-02T22:21:13.675 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2023-12-02T22:21:13.675 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlzZInDhAArPTHKxaAZajR3uOAIXPXZg== 2023-12-02T22:21:13.675 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-12-02T22:21:13.675 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDQrWtlg9phBRAAdpJjGP2okmSOKekUUdxOvw== 2023-12-02T22:21:13.675 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-12-02T22:21:13.675 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDTrWtlRDuGAxAAgFzhb5VDX52IRXYOlaLW5g== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDWrWtlQX16ARAA7SQyKgJ30XqdB7+mvF0iXQ== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtleo4XERAAVMbNGOuDZ+OzOUl/xAHpFQ== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtl+J7kFBAA2T5DiByIViZaLuMcYnF4qw== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlgMjGGhAAZ7tw8Urb3Zp8HHJ0YQAfcQ== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlBUmPIBAAfWdsH1khMXQ/zLNbfxfQOg== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtloktiJhAA9f06bZUxAdbj5Gs/GDAwDg== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtl9wssLBAAQt31/qALUcaHrsBzkh9TKQ== 2023-12-02T22:21:13.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.4] 2023-12-02T22:21:13.677 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlelf2MRAAF3TsQGYCNbmLMwEVHOJitA== 2023-12-02T22:21:13.677 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.5] 2023-12-02T22:21:13.677 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlJUnsNxAAdLED2SfB4H9nULZiabSodg== 2023-12-02T22:21:13.677 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.6] 2023-12-02T22:21:13.677 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDPrWtl4iQcAhAAmZBWYvJcMiMSLUQ2wbtytQ== 2023-12-02T22:21:13.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-02T22:21:13.777 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T22:21:13.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T22:21:13.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T22:21:13.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:13.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.046 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.214 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.327 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T22:21:14.438 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-02T22:21:14.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-02T22:21:14.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-02T22:21:14.565 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-02T22:21:14.597 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T22:21:14.653 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-02T22:21:14.653 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-02T22:21:14.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-02T22:21:14.709 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-02T22:21:14.709 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-02T22:21:14.710 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-02T22:21:14.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-02T22:21:14.713 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-02T22:21:14.713 INFO:tasks.ceph:Setting crush tunables to default 2023-12-02T22:21:14.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-02T22:21:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:adjusted tunables profile to default 2023-12-02T22:21:14.912 INFO:tasks.ceph:check_enable_crimson: False 2023-12-02T22:21:14.912 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-02T22:21:14.912 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:14.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-02T22:21:14.941 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:14.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-02T22:21:15.009 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:21:15.009 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-02T22:21:15.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd new 28c765d0-153b-42a5-8c7d-9e7eab6c4b27 0 2023-12-02T22:21:15.181 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.176+0000 7fa5c97f1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-02T22:21:15.225 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.224+0000 7fa5c97f1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-02T22:21:15.293 INFO:teuthology.orchestra.run.smithi170.stdout:0 2023-12-02T22:21:15.305 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd new 9b6610ce-b97c-42f3-8b70-31dab85e509c 1 2023-12-02T22:21:15.479 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-12-02T22:21:15.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd new 54597968-f0e2-4359-bf21-d8d4a16ed3cb 2 2023-12-02T22:21:15.636 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.634+0000 7fa5c97f1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-02T22:21:15.670 INFO:teuthology.orchestra.run.smithi170.stdout:2 2023-12-02T22:21:15.681 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-02T22:21:15.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-02T22:21:15.682 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.680+0000 7fa5c97f1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-02T22:21:15.684 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-02T22:21:15.684 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-02T22:21:15.684 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-02T22:21:15.687 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-02T22:21:15.687 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-02T22:21:15.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-02T22:21:15.689 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-02T22:21:15.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-02T22:21:15.726 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.725+0000 7fa5c97f1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-02T22:21:15.780 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.778+0000 7fa5c97f1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-02T22:21:15.823 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.822+0000 7fa5c97f1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-02T22:21:15.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:15.859 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":5,"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","created":"2023-12-02T22:21:14.817811+0000","modified":"2023-12-02T22:21:15.667774+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T22:21:15.667774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28c765d0-153b-42a5-8c7d-9e7eab6c4b27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9b6610ce-b97c-42f3-8b70-31dab85e509c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"54597968-f0e2-4359-bf21-d8d4a16ed3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:21:15.870 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-02T22:21:15.870 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-02T22:21:15.928 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.925+0000 7fa5c97f1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-02T22:21:15.982 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:15.981+0000 7fa5c97f1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-02T22:21:16.026 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.024+0000 7fa5c97f1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-02T22:21:16.077 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.075+0000 7fa5c97f1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-02T22:21:16.122 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.119+0000 7fa5c97f1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-02T22:21:16.175 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-02T22:21:16.279 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:21:16.276+0000 7f2e3380d500 -1 Falling back to public interface 2023-12-02T22:21:16.282 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:21:16.280+0000 7f4a39816500 -1 Falling back to public interface 2023-12-02T22:21:16.283 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:21:16.281+0000 7f316c3ad500 -1 Falling back to public interface 2023-12-02T22:21:16.336 INFO:teuthology.misc.health.smithi170.stdout: 2023-12-02T22:21:16.336 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":5,"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","created":"2023-12-02T22:21:14.817811+0000","modified":"2023-12-02T22:21:15.667774+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T22:21:15.667774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28c765d0-153b-42a5-8c7d-9e7eab6c4b27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9b6610ce-b97c-42f3-8b70-31dab85e509c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"54597968-f0e2-4359-bf21-d8d4a16ed3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:21:16.344 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T22:21:16.452 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.451+0000 7fa5c97f1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-02T22:21:16.592 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.590+0000 7fa5c97f1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-02T22:21:16.688 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.686+0000 7fa5c97f1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-02T22:21:16.733 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.731+0000 7fa5c97f1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-02T22:21:16.832 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.831+0000 7fa5c97f1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-02T22:21:16.928 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:16.926+0000 7fa5c97f1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-02T22:21:17.071 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:17.070+0000 7fa5c97f1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-02T22:21:17.117 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:17.114+0000 7fa5c97f1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-02T22:21:17.225 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:17.224+0000 7fa5c97f1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-02T22:21:17.853 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:17.851+0000 7fa5c97f1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-02T22:21:18.129 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:18.127+0000 7fa5c97f1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-02T22:21:18.181 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:18.178+0000 7fa5c97f1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-02T22:21:18.288 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:18.283+0000 7fa5c97f1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-02T22:21:19.294 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T22:21:19.292+0000 7fa56b33c640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-02T22:21:19.715 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:21:19.713+0000 7f2e3380d500 -1 osd.2 0 log_to_monitors true 2023-12-02T22:21:19.762 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:21:19.760+0000 7f4a39816500 -1 osd.1 0 log_to_monitors true 2023-12-02T22:21:19.770 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:21:19.766+0000 7f316c3ad500 -1 osd.0 0 log_to_monitors true 2023-12-02T22:21:22.504 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:21:22.502+0000 7f4a3499e640 -1 osd.1 0 waiting for initial osdmap 2023-12-02T22:21:22.514 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:21:22.512+0000 7f2e301cd640 -1 osd.2 0 waiting for initial osdmap 2023-12-02T22:21:22.516 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:21:22.514+0000 7f4a2cfd7640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T22:21:22.532 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:21:22.531+0000 7f2e26fce640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T22:21:22.553 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:21:22.551+0000 7f3169352640 -1 osd.0 0 waiting for initial osdmap 2023-12-02T22:21:22.568 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:21:22.566+0000 7f3160153640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T22:21:22.654 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-02T22:21:22.922 INFO:teuthology.misc.health.smithi170.stdout: 2023-12-02T22:21:22.922 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":7,"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","created":"2023-12-02T22:21:14.817811+0000","modified":"2023-12-02T22:21:21.290817+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T22:21:15.667774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28c765d0-153b-42a5-8c7d-9e7eab6c4b27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9b6610ce-b97c-42f3-8b70-31dab85e509c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"54597968-f0e2-4359-bf21-d8d4a16ed3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:21:22.934 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T22:21:29.240 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-02T22:21:29.521 INFO:teuthology.misc.health.smithi170.stdout: 2023-12-02T22:21:29.523 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":11,"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","created":"2023-12-02T22:21:14.817811+0000","modified":"2023-12-02T22:21:26.515978+0000","last_up_change":"2023-12-02T22:21:23.514081+0000","last_in_change":"2023-12-02T22:21:15.667774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:21:24.330958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28c765d0-153b-42a5-8c7d-9e7eab6c4b27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6813","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6815","nonce":1523262309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6818","nonce":1523262309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6823","nonce":1523262309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6821","nonce":1523262309}]},"public_addr":"172.21.15.170:6815/1523262309","cluster_addr":"172.21.15.170:6818/1523262309","heartbeat_back_addr":"172.21.15.170:6823/1523262309","heartbeat_front_addr":"172.21.15.170:6821/1523262309","state":["exists","up"]},{"osd":1,"uuid":"9b6610ce-b97c-42f3-8b70-31dab85e509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6809","nonce":2583416787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6811","nonce":2583416787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6817","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6819","nonce":2583416787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6814","nonce":2583416787}]},"public_addr":"172.21.15.170:6809/2583416787","cluster_addr":"172.21.15.170:6811/2583416787","heartbeat_back_addr":"172.21.15.170:6819/2583416787","heartbeat_front_addr":"172.21.15.170:6814/2583416787","state":["exists","up"]},{"osd":2,"uuid":"54597968-f0e2-4359-bf21-d8d4a16ed3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6801","nonce":461241410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6803","nonce":461241410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6807","nonce":461241410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6805","nonce":461241410}]},"public_addr":"172.21.15.170:6801/461241410","cluster_addr":"172.21.15.170:6803/461241410","heartbeat_back_addr":"172.21.15.170:6807/461241410","heartbeat_front_addr":"172.21.15.170:6805/461241410","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.731103+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.807011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.752524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:21:29.532 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-02T22:21:29.533 INFO:tasks.ceph:Creating RBD pool 2023-12-02T22:21:29.533 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-02T22:21:30.782 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'rbd' created 2023-12-02T22:21:30.800 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-02T22:21:31.785 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-02T22:21:31.808 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-02T22:21:31.808 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-02T22:21:31.809 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-02T22:21:31.809 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-02T22:21:31.809 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-02T22:21:32.141 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:32.153 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3247171133},{"type":"v1","addr":"172.21.15.12:6827","nonce":3247171133}]},"active_addr":"172.21.15.12:6827/3247171133","active_change":"2023-12-02T22:21:18.285935+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2689422068}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2815503449}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":4284460664}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":402163858}]}]} 2023-12-02T22:21:32.168 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-02T22:21:32.168 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-02T22:21:32.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-02T22:21:32.448 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:32.448 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":13,"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","created":"2023-12-02T22:21:14.817811+0000","modified":"2023-12-02T22:21:31.781867+0000","last_up_change":"2023-12-02T22:21:23.514081+0000","last_in_change":"2023-12-02T22:21:15.667774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:21:24.330958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T22:21:29.831466+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28c765d0-153b-42a5-8c7d-9e7eab6c4b27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6813","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6815","nonce":1523262309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6818","nonce":1523262309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6823","nonce":1523262309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6821","nonce":1523262309}]},"public_addr":"172.21.15.170:6815/1523262309","cluster_addr":"172.21.15.170:6818/1523262309","heartbeat_back_addr":"172.21.15.170:6823/1523262309","heartbeat_front_addr":"172.21.15.170:6821/1523262309","state":["exists","up"]},{"osd":1,"uuid":"9b6610ce-b97c-42f3-8b70-31dab85e509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6809","nonce":2583416787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6811","nonce":2583416787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6817","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6819","nonce":2583416787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6814","nonce":2583416787}]},"public_addr":"172.21.15.170:6809/2583416787","cluster_addr":"172.21.15.170:6811/2583416787","heartbeat_back_addr":"172.21.15.170:6819/2583416787","heartbeat_front_addr":"172.21.15.170:6814/2583416787","state":["exists","up"]},{"osd":2,"uuid":"54597968-f0e2-4359-bf21-d8d4a16ed3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6801","nonce":461241410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6803","nonce":461241410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6807","nonce":461241410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6805","nonce":461241410}]},"public_addr":"172.21.15.170:6801/461241410","cluster_addr":"172.21.15.170:6803/461241410","heartbeat_back_addr":"172.21.15.170:6807/461241410","heartbeat_front_addr":"172.21.15.170:6805/461241410","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.731103+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.807011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.752524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:21:32.461 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-02T22:21:32.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-02T22:21:32.740 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:32.740 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":13,"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","created":"2023-12-02T22:21:14.817811+0000","modified":"2023-12-02T22:21:31.781867+0000","last_up_change":"2023-12-02T22:21:23.514081+0000","last_in_change":"2023-12-02T22:21:15.667774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:21:24.330958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T22:21:29.831466+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28c765d0-153b-42a5-8c7d-9e7eab6c4b27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6813","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6815","nonce":1523262309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6818","nonce":1523262309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6823","nonce":1523262309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1523262309},{"type":"v1","addr":"172.21.15.170:6821","nonce":1523262309}]},"public_addr":"172.21.15.170:6815/1523262309","cluster_addr":"172.21.15.170:6818/1523262309","heartbeat_back_addr":"172.21.15.170:6823/1523262309","heartbeat_front_addr":"172.21.15.170:6821/1523262309","state":["exists","up"]},{"osd":1,"uuid":"9b6610ce-b97c-42f3-8b70-31dab85e509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6809","nonce":2583416787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6811","nonce":2583416787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6817","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6819","nonce":2583416787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2583416787},{"type":"v1","addr":"172.21.15.170:6814","nonce":2583416787}]},"public_addr":"172.21.15.170:6809/2583416787","cluster_addr":"172.21.15.170:6811/2583416787","heartbeat_back_addr":"172.21.15.170:6819/2583416787","heartbeat_front_addr":"172.21.15.170:6814/2583416787","state":["exists","up"]},{"osd":2,"uuid":"54597968-f0e2-4359-bf21-d8d4a16ed3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6801","nonce":461241410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6803","nonce":461241410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6807","nonce":461241410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":461241410},{"type":"v1","addr":"172.21.15.170:6805","nonce":461241410}]},"public_addr":"172.21.15.170:6801/461241410","cluster_addr":"172.21.15.170:6803/461241410","heartbeat_back_addr":"172.21.15.170:6807/461241410","heartbeat_front_addr":"172.21.15.170:6805/461241410","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.731103+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.807011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:21:20.752524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T22:21:32.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-02T22:21:32.753 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-02T22:21:32.753 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-02T22:21:32.893 INFO:teuthology.orchestra.run.smithi170.stdout:34359738374 2023-12-02T22:21:32.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-02T22:21:32.895 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2023-12-02T22:21:32.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-02T22:21:32.896 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2023-12-02T22:21:32.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-02T22:21:33.238 INFO:teuthology.orchestra.run.smithi170.stdout:34359738372 2023-12-02T22:21:33.249 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2023-12-02T22:21:33.250 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2023-12-02T22:21:33.252 INFO:teuthology.orchestra.run.smithi170.stdout:34359738372 2023-12-02T22:21:33.264 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-02T22:21:33.265 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2023-12-02T22:21:34.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-02T22:21:34.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-02T22:21:34.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-02T22:21:34.559 INFO:teuthology.orchestra.run.smithi170.stdout:34359738372 2023-12-02T22:21:34.572 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2023-12-02T22:21:34.594 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2023-12-02T22:21:34.598 INFO:teuthology.orchestra.run.smithi170.stdout:34359738372 2023-12-02T22:21:34.607 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-02T22:21:34.612 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2023-12-02T22:21:35.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-02T22:21:35.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-02T22:21:35.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-02T22:21:35.906 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2023-12-02T22:21:35.919 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2023-12-02T22:21:35.919 DEBUG:teuthology.parallel:result is None 2023-12-02T22:21:35.921 INFO:teuthology.orchestra.run.smithi170.stdout:34359738374 2023-12-02T22:21:35.934 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2023-12-02T22:21:35.934 DEBUG:teuthology.parallel:result is None 2023-12-02T22:21:35.945 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2023-12-02T22:21:35.957 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.1 2023-12-02T22:21:35.957 DEBUG:teuthology.parallel:result is None 2023-12-02T22:21:35.957 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-02T22:21:35.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:21:36.275 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:36.275 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:21:36.284 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-02T22:21:34.292602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95048,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020472,"statfs":{"total":289910292480,"available":289812963328,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000878"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788692+0000","last_change":"2023-12-02T22:21:31.791339+0000","last_active":"2023-12-02T22:21:32.788692+0000","last_peered":"2023-12-02T22:21:32.788692+0000","last_clean":"2023-12-02T22:21:32.788692+0000","last_became_active":"2023-12-02T22:21:31.790896+0000","last_became_peered":"2023-12-02T22:21:31.790896+0000","last_unstale":"2023-12-02T22:21:32.788692+0000","last_undegraded":"2023-12-02T22:21:32.788692+0000","last_fullsized":"2023-12-02T22:21:32.788692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:02:11.666694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788573+0000","last_change":"2023-12-02T22:21:31.791989+0000","last_active":"2023-12-02T22:21:32.788573+0000","last_peered":"2023-12-02T22:21:32.788573+0000","last_clean":"2023-12-02T22:21:32.788573+0000","last_became_active":"2023-12-02T22:21:31.791718+0000","last_became_peered":"2023-12-02T22:21:31.791718+0000","last_unstale":"2023-12-02T22:21:32.788573+0000","last_undegraded":"2023-12-02T22:21:32.788573+0000","last_fullsized":"2023-12-02T22:21:32.788573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:29:35.278030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788679+0000","last_change":"2023-12-02T22:21:31.791969+0000","last_active":"2023-12-02T22:21:32.788679+0000","last_peered":"2023-12-02T22:21:32.788679+0000","last_clean":"2023-12-02T22:21:32.788679+0000","last_became_active":"2023-12-02T22:21:31.790960+0000","last_became_peered":"2023-12-02T22:21:31.790960+0000","last_unstale":"2023-12-02T22:21:32.788679+0000","last_undegraded":"2023-12-02T22:21:32.788679+0000","last_fullsized":"2023-12-02T22:21:32.788679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:55:07.687172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788915+0000","last_change":"2023-12-02T22:21:31.791542+0000","last_active":"2023-12-02T22:21:32.788915+0000","last_peered":"2023-12-02T22:21:32.788915+0000","last_clean":"2023-12-02T22:21:32.788915+0000","last_became_active":"2023-12-02T22:21:31.791180+0000","last_became_peered":"2023-12-02T22:21:31.791180+0000","last_unstale":"2023-12-02T22:21:32.788915+0000","last_undegraded":"2023-12-02T22:21:32.788915+0000","last_fullsized":"2023-12-02T22:21:32.788915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:52:44.080464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.793019+0000","last_change":"2023-12-02T22:21:31.790894+0000","last_active":"2023-12-02T22:21:31.793019+0000","last_peered":"2023-12-02T22:21:31.793019+0000","last_clean":"2023-12-02T22:21:31.793019+0000","last_became_active":"2023-12-02T22:21:31.790588+0000","last_became_peered":"2023-12-02T22:21:31.790588+0000","last_unstale":"2023-12-02T22:21:31.793019+0000","last_undegraded":"2023-12-02T22:21:31.793019+0000","last_fullsized":"2023-12-02T22:21:31.793019+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:14:56.740981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.793335+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:21:31.793335+0000","last_peered":"2023-12-02T22:21:31.793335+0000","last_clean":"2023-12-02T22:21:31.793335+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:21:31.793335+0000","last_undegraded":"2023-12-02T22:21:31.793335+0000","last_fullsized":"2023-12-02T22:21:31.793335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:09:34.717860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.792587+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:21:31.792587+0000","last_peered":"2023-12-02T22:21:31.792587+0000","last_clean":"2023-12-02T22:21:31.792587+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:21:31.792587+0000","last_undegraded":"2023-12-02T22:21:31.792587+0000","last_fullsized":"2023-12-02T22:21:31.792587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:00:39.521916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788557+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:21:32.788557+0000","last_peered":"2023-12-02T22:21:32.788557+0000","last_clean":"2023-12-02T22:21:32.788557+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:21:32.788557+0000","last_undegraded":"2023-12-02T22:21:32.788557+0000","last_fullsized":"2023-12-02T22:21:32.788557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.789248+0000","last_change":"2023-12-02T22:21:25.521340+0000","last_active":"2023-12-02T22:21:31.789248+0000","last_peered":"2023-12-02T22:21:31.789248+0000","last_clean":"2023-12-02T22:21:31.789248+0000","last_became_active":"2023-12-02T22:21:25.520327+0000","last_became_peered":"2023-12-02T22:21:25.520327+0000","last_unstale":"2023-12-02T22:21:31.789248+0000","last_undegraded":"2023-12-02T22:21:31.789248+0000","last_fullsized":"2023-12-02T22:21:31.789248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:10:12.976248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:21:36.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:21:36.579 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:36.580 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:21:36.592 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-02T22:21:36.293047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95064,"kb_used_data":1512,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020456,"statfs":{"total":289910292480,"available":289812946944,"internally_reserved":0,"allocated":1548288,"data_stored":1281685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788692+0000","last_change":"2023-12-02T22:21:31.791339+0000","last_active":"2023-12-02T22:21:32.788692+0000","last_peered":"2023-12-02T22:21:32.788692+0000","last_clean":"2023-12-02T22:21:32.788692+0000","last_became_active":"2023-12-02T22:21:31.790896+0000","last_became_peered":"2023-12-02T22:21:31.790896+0000","last_unstale":"2023-12-02T22:21:32.788692+0000","last_undegraded":"2023-12-02T22:21:32.788692+0000","last_fullsized":"2023-12-02T22:21:32.788692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:02:11.666694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788573+0000","last_change":"2023-12-02T22:21:31.791989+0000","last_active":"2023-12-02T22:21:32.788573+0000","last_peered":"2023-12-02T22:21:32.788573+0000","last_clean":"2023-12-02T22:21:32.788573+0000","last_became_active":"2023-12-02T22:21:31.791718+0000","last_became_peered":"2023-12-02T22:21:31.791718+0000","last_unstale":"2023-12-02T22:21:32.788573+0000","last_undegraded":"2023-12-02T22:21:32.788573+0000","last_fullsized":"2023-12-02T22:21:32.788573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:29:35.278030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788679+0000","last_change":"2023-12-02T22:21:31.791969+0000","last_active":"2023-12-02T22:21:32.788679+0000","last_peered":"2023-12-02T22:21:32.788679+0000","last_clean":"2023-12-02T22:21:32.788679+0000","last_became_active":"2023-12-02T22:21:31.790960+0000","last_became_peered":"2023-12-02T22:21:31.790960+0000","last_unstale":"2023-12-02T22:21:32.788679+0000","last_undegraded":"2023-12-02T22:21:32.788679+0000","last_fullsized":"2023-12-02T22:21:32.788679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:55:07.687172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788915+0000","last_change":"2023-12-02T22:21:31.791542+0000","last_active":"2023-12-02T22:21:32.788915+0000","last_peered":"2023-12-02T22:21:32.788915+0000","last_clean":"2023-12-02T22:21:32.788915+0000","last_became_active":"2023-12-02T22:21:31.791180+0000","last_became_peered":"2023-12-02T22:21:31.791180+0000","last_unstale":"2023-12-02T22:21:32.788915+0000","last_undegraded":"2023-12-02T22:21:32.788915+0000","last_fullsized":"2023-12-02T22:21:32.788915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:52:44.080464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:33.171399+0000","last_change":"2023-12-02T22:21:31.790894+0000","last_active":"2023-12-02T22:21:33.171399+0000","last_peered":"2023-12-02T22:21:33.171399+0000","last_clean":"2023-12-02T22:21:33.171399+0000","last_became_active":"2023-12-02T22:21:31.790588+0000","last_became_peered":"2023-12-02T22:21:31.790588+0000","last_unstale":"2023-12-02T22:21:33.171399+0000","last_undegraded":"2023-12-02T22:21:33.171399+0000","last_fullsized":"2023-12-02T22:21:33.171399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:33:49.494713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.793335+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:21:31.793335+0000","last_peered":"2023-12-02T22:21:31.793335+0000","last_clean":"2023-12-02T22:21:31.793335+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:21:31.793335+0000","last_undegraded":"2023-12-02T22:21:31.793335+0000","last_fullsized":"2023-12-02T22:21:31.793335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:09:34.717860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.792587+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:21:31.792587+0000","last_peered":"2023-12-02T22:21:31.792587+0000","last_clean":"2023-12-02T22:21:31.792587+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:21:31.792587+0000","last_undegraded":"2023-12-02T22:21:31.792587+0000","last_fullsized":"2023-12-02T22:21:31.792587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:00:39.521916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T22:21:32.788557+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:21:32.788557+0000","last_peered":"2023-12-02T22:21:32.788557+0000","last_clean":"2023-12-02T22:21:32.788557+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:21:32.788557+0000","last_undegraded":"2023-12-02T22:21:32.788557+0000","last_fullsized":"2023-12-02T22:21:32.788557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T22:21:31.789248+0000","last_change":"2023-12-02T22:21:25.521340+0000","last_active":"2023-12-02T22:21:31.789248+0000","last_peered":"2023-12-02T22:21:31.789248+0000","last_clean":"2023-12-02T22:21:31.789248+0000","last_became_active":"2023-12-02T22:21:25.520327+0000","last_became_peered":"2023-12-02T22:21:25.520327+0000","last_unstale":"2023-12-02T22:21:31.789248+0000","last_undegraded":"2023-12-02T22:21:31.789248+0000","last_fullsized":"2023-12-02T22:21:31.789248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:10:12.976248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:21:36.593 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-02T22:21:36.593 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-02T22:21:36.593 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-02T22:21:36.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-02T22:21:36.909 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:21:36.909 INFO:teuthology.orchestra.run.smithi170.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-02T22:21:36.925 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-02T22:21:36.928 INFO:teuthology.run_tasks:Running task exec... 2023-12-02T22:21:36.939 INFO:teuthology.task.exec:Executing custom commands... 2023-12-02T22:21:36.945 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T22:21:36.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:37.297 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror] 2023-12-02T22:21:37.297 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCprWtlkkc4GhAAkjQNNcwngaveBH3hHGq9Ow== 2023-12-02T22:21:37.298 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-02T22:21:37.298 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:37.298 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror 2023-12-02T22:21:37.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:37.666 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2023-12-02T22:21:37.667 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCprWtlSH7ZHRAAOMIGw73UjsDQ69Xnp+AwMg== 2023-12-02T22:21:37.667 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:37.667 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:37.667 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.0 2023-12-02T22:21:37.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:38.039 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2023-12-02T22:21:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCprWtlr03HIxAA6XNUrBnS4oFDS52XUdQCAg== 2023-12-02T22:21:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:38.040 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.1 2023-12-02T22:21:38.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:38.409 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2023-12-02T22:21:38.410 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCprWtldNGXKRAArjrGzbcFao1CydNSNPOWyQ== 2023-12-02T22:21:38.410 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:38.410 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:38.410 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.2 2023-12-02T22:21:38.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:38.779 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2023-12-02T22:21:38.779 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCprWtlNptiLxAASflO3FjF+0v9lrZXnIVR6w== 2023-12-02T22:21:38.779 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:38.780 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:38.780 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.3 2023-12-02T22:21:38.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:39.144 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror] 2023-12-02T22:21:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtl+J7kFBAA2T5DiByIViZaLuMcYnF4qw== 2023-12-02T22:21:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-02T22:21:39.145 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:39.145 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror 2023-12-02T22:21:39.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:39.523 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2023-12-02T22:21:39.523 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlgMjGGhAAZ7tw8Urb3Zp8HHJ0YQAfcQ== 2023-12-02T22:21:39.524 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:39.524 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:39.524 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.0 2023-12-02T22:21:39.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:39.890 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2023-12-02T22:21:39.890 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtlBUmPIBAAfWdsH1khMXQ/zLNbfxfQOg== 2023-12-02T22:21:39.890 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:39.890 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:39.890 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.1 2023-12-02T22:21:39.903 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:40.256 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2023-12-02T22:21:40.256 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtloktiJhAA9f06bZUxAdbj5Gs/GDAwDg== 2023-12-02T22:21:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:40.257 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.2 2023-12-02T22:21:40.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T22:21:40.624 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2023-12-02T22:21:40.624 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDOrWtl9wssLBAAQt31/qALUcaHrsBzkh9TKQ== 2023-12-02T22:21:40.624 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2023-12-02T22:21:40.624 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2023-12-02T22:21:40.624 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.3 2023-12-02T22:21:40.638 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T22:21:40.657 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T22:21:40.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T22:21:40.662 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T22:21:40.687 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T22:21:40.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T22:21:40.697 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T22:21:40.710 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T22:21:40.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T22:21:40.735 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T22:21:40.775 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T22:21:40.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T22:21:40.800 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:21:40.810 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:21:40.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:21:40.832 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T22:21:40.894 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T22:21:40.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T22:21:40.899 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:40.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T22:21:40.967 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T22:21:41.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T22:21:41.011 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T22:21:41.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T22:21:41.041 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T22:21:41.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T22:21:41.066 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-02T22:21:41.086 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701555701 2023-12-02T22:21:41.086 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-02T22:21:41.086 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-02T22:21:41.097 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701555701 2023-12-02T22:21:41.097 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-02T22:21:41.101 INFO:teuthology.run_tasks:Running task workunit... 2023-12-02T22:21:41.143 INFO:tasks.workunit:Pulling workunits from ref 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:21:41.144 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-02T22:21:41.144 DEBUG:teuthology.orchestra.run.smithi170:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T22:21:41.144 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-02T22:21:41.145 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-12-02T22:21:41.145 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-02T22:21:41.145 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-12-02T22:21:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T22:21:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-02T22:21:41.162 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T22:21:41.216 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T22:21:41.217 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-02T22:21:41.271 INFO:tasks.workunit:timeout=6h 2023-12-02T22:21:41.271 INFO:tasks.workunit:cleanup=True 2023-12-02T22:21:41.272 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 2023-12-02T22:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-02T22:22:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Note: switching to '0c523c70ca7aa7b49aa5b461902b17d9e90c1f95'. 2023-12-02T22:22:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:state without impacting any branches by switching back to a branch. 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: git switch -c 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Or undo this operation with: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: git switch - 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-02T22:22:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:22:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:HEAD is now at 0c523c70ca7 Merge branch 'wip-63654' of https://github.com/idryomov/ceph into HEAD 2023-12-02T22:22:32.439 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-02T22:22:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-02T22:22:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-02T22:22:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-02T22:22:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-02T22:22:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-02T22:22:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-02T22:22:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-02T22:22:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-02T22:22:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-02T22:22:33.437 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-02T22:22:33.438 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-02T22:22:33.493 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-02T22:22:33.495 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-02T22:22:33.495 DEBUG:teuthology.orchestra.run.smithi170:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c523c70ca7aa7b49aa5b461902b17d9e90c1f95 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-02T22:22:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ IMAGE_COUNT=50 2023-12-02T22:22:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ export LOCKDEP=0 2023-12-02T22:22:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ LOCKDEP=0 2023-12-02T22:22:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-02T22:22:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-02T22:22:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ type xmlstarlet 2023-12-02T22:22:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ XMLSTARLET=xmlstarlet 2023-12-02T22:22:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ RBD_MIRROR_INSTANCES=4 2023-12-02T22:22:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CLUSTER1=cluster1 2023-12-02T22:22:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CLUSTER2=cluster2 2023-12-02T22:22:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ PEER_CLUSTER_SUFFIX= 2023-12-02T22:22:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ POOL=mirror 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ PARENT_POOL=mirror_parent 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ NS1=ns1 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ NS2=ns2 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ TEMPDIR= 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ID=mirror 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-12-02T22:22:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ RBD_MIRROR_MODE=snapshot 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ MIRROR_POOL_MODE=image 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' snapshot = snapshot ']' 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ MIRROR_POOL_MODE=image 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS='--id mirror' 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ LAST_MIRROR_INSTANCE=3 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-02T22:22:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-02T22:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-02T22:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_BIN=. 2023-12-02T22:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_SRC=. 2023-12-02T22:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' -e CMakeCache.txt ']' 2023-12-02T22:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 -gt 0 ']' 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local c 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup_tempdir 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mktemp -d 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ TEMPDIR=/tmp/tmp.cQPVpMBksv 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z 1 ']' 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup_pools cluster1 cluster2 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster2 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_map_file 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_addr 2023-12-02T22:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local admin_key_file 2023-12-02T22:22:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid 2023-12-02T22:22:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-02T22:22:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' already exists 2023-12-02T22:22:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-02T22:22:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' already exists 2023-12-02T22:22:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 pool init mirror 2023-12-02T22:22:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-12-02T22:22:41.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5742145503890078) > weight (0.25) 2023-12-02T22:22:41.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5266092545263049) > weight (0.25) 2023-12-02T22:22:41.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.39085122021904517) > weight (0.25) 2023-12-02T22:22:41.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5459572384367972) > weight (0.25) 2023-12-02T22:22:41.146 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-12-02T22:22:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-02T22:22:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-02T22:22:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-02T22:22:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-02T22:22:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-02T22:22:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-02T22:22:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2023-12-02T22:22:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2023-12-02T22:22:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster1 mirror cluster2 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster2 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster2 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T22:22:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:22:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-02T22:22:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=b4bebeea-8d50-4b0f-b07c-3731ab267316 2023-12-02T22:22:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:22:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:22:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T22:22:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2023-12-02T22:22:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror_parent 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster2 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster2 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T22:22:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:22:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=a40e3209-1dee-4bca-a4df-356eca9a6fa7 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup_pools cluster2 cluster1 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster1 2023-12-02T22:22:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_map_file 2023-12-02T22:22:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_addr 2023-12-02T22:22:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local admin_key_file 2023-12-02T22:22:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid 2023-12-02T22:22:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-02T22:22:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' already exists 2023-12-02T22:22:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-02T22:22:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' already exists 2023-12-02T22:22:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 pool init mirror 2023-12-02T22:22:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-12-02T22:22:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:22:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-02T22:22:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-02T22:22:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-02T22:22:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-02T22:22:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-02T22:22:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-02T22:22:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2023-12-02T22:22:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster2 mirror cluster1 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster1 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster1 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:22:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-02T22:22:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=0ca71c52-11e7-40b4-937b-96bd49fcca8b 2023-12-02T22:22:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:22:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:22:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T22:22:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2023-12-02T22:22:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror_parent 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster1 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster1 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T22:22:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:22:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=91241553-4190-45ba-be25-7142ffbdf931 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:22:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirrors cluster1 2023-12-02T22:22:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 3 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1:0 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1:1 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:22:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1:2 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:2 2023-12-02T22:22:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:2 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=2 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=2 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1:3 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:3 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:3 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=3 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=3 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:22:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirrors cluster2 2023-12-02T22:22:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 3 2023-12-02T22:22:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster2:0 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:0 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster2:0 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster2 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2023-12-02T22:22:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster2:0 ']' 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster2 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster2:1 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:1 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster2:1 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster2 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2023-12-02T22:22:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster2:1 ']' 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster2 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster2:2 2023-12-02T22:22:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster2:2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 2 = cluster2:2 ']' 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:22:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster2:3 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:3 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster2:3 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:22:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster2 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=3 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 3 = cluster2:3 ']' 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster2 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=3 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:22:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:22:50 TEST: add image and test replay after client crashes 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:22:50 TEST: add image and test replay after client crashes 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=test 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror test 512M 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:22:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:22:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:22:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:22:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:22:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-02T22:22:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:22:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10aaf86e67ac 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10aaf86e67ac 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:22:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:22:50 2023 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:22:50 2023 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:22:50 2023 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:22:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:22:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:22:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:22:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:22:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:22:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:22:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:22:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:22:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:22:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:22:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:22:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:22:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:22:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:22:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:22:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:22:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:22:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:22:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:22:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T22:22:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:22:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:22:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:22:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:22:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:22:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T22:22:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:22:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:22:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:22:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T22:22:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:22:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 1 2023-12-02T22:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:22:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:22:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:22:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:22:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:22:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:22:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:22:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T22:22:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:22:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:22:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:22:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T22:22:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T22:22:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:22:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:22:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:22:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T22:22:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T22:22:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:22:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:22:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:22:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:22:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T22:22:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:22:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 2 2023-12-02T22:22:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:22:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:22:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:22:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:22:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:22:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:22:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:22:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:22:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:22:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:22:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:22:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T22:22:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:22:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:22:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:22:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:22:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:22:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T22:22:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:22:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:22:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:22:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:22:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 4 2023-12-02T22:23:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:23:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:23:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:23:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:23:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:23:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:23:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:23:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:23:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:23:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T22:23:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T22:23:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:23:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:23:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:23:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:23:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:23:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:23:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:23:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:23:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T22:23:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:23:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:23:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:23:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:23:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:23:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T22:23:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 8 2023-12-02T22:23:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:23:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:23:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:23:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:23:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:23:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T22:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:23:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:23:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:23:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:23:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T22:23:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T22:23:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:23:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:23:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:23:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:23:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:23:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:23:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:23:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:23:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T22:23:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:23:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:23:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:23:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:23:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:23:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T22:23:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T22:23:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:23:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:23:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 8 2023-12-02T22:23:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:23:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:23:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:23:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:23:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:23:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:23:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:23:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:23:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:23:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:23:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:23:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:23:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:23:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:23:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:23:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name= 2023-12-02T22:23:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2023-12-02T22:23:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:23:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:23:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:23:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:23:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:23:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:23:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:23:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 20135936 bytes 2023-12-02T22:23:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 60873728 bytes 2023-12-02T22:23:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 125926400 bytes 2023-12-02T22:23:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 168742912 bytes 2023-12-02T22:23:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 185916416 bytes 2023-12-02T22:23:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 247085056 bytes 2023-12-02T22:23:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 275121152 bytes 2023-12-02T22:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 275737600 bytes 2023-12-02T22:23:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 334513152 bytes 2023-12-02T22:23:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 399657984 bytes 2023-12-02T22:23:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 459018240 bytes 2023-12-02T22:23:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 483362816 bytes 2023-12-02T22:23:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 512015360 bytes 2023-12-02T22:23:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 528615424 bytes 2023-12-02T22:23:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 561625088 bytes 2023-12-02T22:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 619735040 bytes 2023-12-02T22:23:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 631027712 bytes 2023-12-02T22:23:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 642573312 bytes 2023-12-02T22:23:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 674783232 bytes 2023-12-02T22:23:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 724725760 bytes 2023-12-02T22:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 779383808 bytes 2023-12-02T22:23:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 858114048 bytes 2023-12-02T22:23:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 858728448 bytes 2023-12-02T22:23:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 937292800 bytes 2023-12-02T22:23:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 985401344 bytes 2023-12-02T22:23:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1050771456 bytes 2023-12-02T22:23:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1125065728 bytes 2023-12-02T22:23:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1190388736 bytes 2023-12-02T22:23:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1289420800 bytes 2023-12-02T22:23:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1333069824 bytes 2023-12-02T22:23:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1368635392 bytes 2023-12-02T22:23:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1382222848 bytes 2023-12-02T22:23:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1394550784 bytes 2023-12-02T22:23:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1426190336 bytes 2023-12-02T22:23:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1435168768 bytes 2023-12-02T22:23:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1475474432 bytes 2023-12-02T22:23:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1516600320 bytes 2023-12-02T22:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1552091136 bytes 2023-12-02T22:23:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1560702976 bytes 2023-12-02T22:23:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1568912384 bytes 2023-12-02T22:23:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1584663552 bytes 2023-12-02T22:23:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1596890112 bytes 2023-12-02T22:23:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1599610880 bytes 2023-12-02T22:23:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1600242688 bytes 2023-12-02T22:23:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1621388288 bytes 2023-12-02T22:23:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1679219712 bytes 2023-12-02T22:23:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1708991488 bytes 2023-12-02T22:23:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1728052224 bytes 2023-12-02T22:23:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1730750464 bytes 2023-12-02T22:23:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1775808512 bytes 2023-12-02T22:23:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1804474368 bytes 2023-12-02T22:23:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1846378496 bytes 2023-12-02T22:23:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1863958528 bytes 2023-12-02T22:23:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 1882820608 bytes 2023-12-02T22:23:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 1904087040 bytes 2023-12-02T22:23:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 1954143232 bytes 2023-12-02T22:23:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 1989215232 bytes 2023-12-02T22:23:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2021109760 bytes 2023-12-02T22:23:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2075461632 bytes 2023-12-02T22:23:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2121263104 bytes 2023-12-02T22:23:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2135044096 bytes 2023-12-02T22:23:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2195567616 bytes 2023-12-02T22:23:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2255383552 bytes 2023-12-02T22:23:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2342449152 bytes 2023-12-02T22:23:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2411613184 bytes 2023-12-02T22:23:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2507603968 bytes 2023-12-02T22:23:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2562939904 bytes 2023-12-02T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2599565312 bytes 2023-12-02T22:23:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2602713088 bytes 2023-12-02T22:23:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2623845376 bytes 2023-12-02T22:23:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2023-12-02T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:23:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:23:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:23:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:23:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:23:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2023-12-02T22:23:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 4 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:23:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:23:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:23:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2023-12-02T22:23:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:23:36.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7047099033720755) > weight (0.25) 2023-12-02T22:23:36.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.29284671057695977) > weight (0.25) 2023-12-02T22:23:36.146 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T22:23:36.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T22:23:36.147 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T22:23:36.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:23:36.146+0000 7fe3c2737640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 90690) UID: 1000 2023-12-02T22:23:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:23:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:23:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:23:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:23:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2023-12-02T22:23:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:23:40.148 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T22:23:40.148 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:23:40.149 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T22:23:40.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T22:23:40.149 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T22:23:40.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T22:23:40.152 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:23:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:23:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:23:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701555770,"remote_snapshot_timestamp":1701555770,"replay_state":"idle"} 2023-12-02T22:23:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:23:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:23:40 2023-12-02T22:23:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:23:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:23:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:23:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:23:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:23:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:23:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701555770,"remote_snapshot_timestamp":1701555770,"replay_state":"idle"} 2023-12-02T22:23:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:23:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:23:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:23:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:23:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:23:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:23:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:23:40' 2023-12-02T22:23:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:23:40' 2023-12-02T22:23:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:23:40' 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:23:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:23:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:23:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:23:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:23:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:23:40' 2023-12-02T22:23:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:23:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:23:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:23:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:23:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap1 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2023-12-02T22:23:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:23:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:23:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:23:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:23:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:23:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:23:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:23:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:23:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:23:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:23:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:23:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:23:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:23:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:23:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:23:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:23:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 6 2023-12-02T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:23:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:23:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:23:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:23:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=6 2023-12-02T22:23:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:23:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:23:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:23:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:23:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:23:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=4 2023-12-02T22:23:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 6 = 4 2023-12-02T22:23:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:23:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:23:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:23:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:23:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=4 2023-12-02T22:23:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 6 = 4 2023-12-02T22:23:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:23:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:23:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:23:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:23:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:23:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:23:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:23:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:23:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:23:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:23:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:23:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:23:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:23:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:23:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 6 = '' 2023-12-02T22:23:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:23:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:24:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:24:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=6 2023-12-02T22:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 6 = 6 2023-12-02T22:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-02T22:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2023-12-02T22:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:24:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:24:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:24:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:24:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:24:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:24:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap1-clean 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1-clean 2023-12-02T22:24:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:24:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:24:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:24:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:24:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2023-12-02T22:24:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:24:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:24:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 14298112 bytes 2023-12-02T22:24:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 31942656 bytes 2023-12-02T22:24:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 72322048 bytes 2023-12-02T22:24:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 72947712 bytes 2023-12-02T22:24:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 102635520 bytes 2023-12-02T22:24:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 103239680 bytes 2023-12-02T22:24:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 153374720 bytes 2023-12-02T22:24:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 234010624 bytes 2023-12-02T22:24:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 260270080 bytes 2023-12-02T22:24:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 319273984 bytes 2023-12-02T22:24:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 342246400 bytes 2023-12-02T22:24:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 423805952 bytes 2023-12-02T22:24:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 474956800 bytes 2023-12-02T22:24:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 523661312 bytes 2023-12-02T22:24:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 549120000 bytes 2023-12-02T22:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 612039680 bytes 2023-12-02T22:24:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 644154368 bytes 2023-12-02T22:24:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 728645632 bytes 2023-12-02T22:24:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 771685376 bytes 2023-12-02T22:24:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 795338752 bytes 2023-12-02T22:24:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 854251520 bytes 2023-12-02T22:24:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 883477504 bytes 2023-12-02T22:24:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 966201344 bytes 2023-12-02T22:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1004873728 bytes 2023-12-02T22:24:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1044087808 bytes 2023-12-02T22:24:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1044678656 bytes 2023-12-02T22:24:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1045285888 bytes 2023-12-02T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1073150976 bytes 2023-12-02T22:24:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1157861376 bytes 2023-12-02T22:24:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1195098112 bytes 2023-12-02T22:24:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1199306752 bytes 2023-12-02T22:24:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1262319616 bytes 2023-12-02T22:24:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1262918656 bytes 2023-12-02T22:24:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1305595904 bytes 2023-12-02T22:24:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1357502464 bytes 2023-12-02T22:24:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1410412544 bytes 2023-12-02T22:24:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1490525184 bytes 2023-12-02T22:24:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1538318336 bytes 2023-12-02T22:24:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1589444608 bytes 2023-12-02T22:24:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1650918400 bytes 2023-12-02T22:24:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1692722176 bytes 2023-12-02T22:24:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1767980032 bytes 2023-12-02T22:24:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1768607744 bytes 2023-12-02T22:24:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1783552000 bytes 2023-12-02T22:24:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1803887616 bytes 2023-12-02T22:24:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1868640256 bytes 2023-12-02T22:24:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1928724480 bytes 2023-12-02T22:24:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1951326208 bytes 2023-12-02T22:24:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2006709248 bytes 2023-12-02T22:24:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2007775232 bytes 2023-12-02T22:24:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2054508544 bytes 2023-12-02T22:24:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2079324160 bytes 2023-12-02T22:24:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2145725440 bytes 2023-12-02T22:24:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2160574464 bytes 2023-12-02T22:24:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2229154816 bytes 2023-12-02T22:24:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2251373568 bytes 2023-12-02T22:24:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2278398976 bytes 2023-12-02T22:24:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2299314176 bytes 2023-12-02T22:24:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2379360256 bytes 2023-12-02T22:24:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2398428160 bytes 2023-12-02T22:24:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2414630912 bytes 2023-12-02T22:24:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2430076928 bytes 2023-12-02T22:24:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2435374080 bytes 2023-12-02T22:24:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2504148992 bytes 2023-12-02T22:24:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2540573696 bytes 2023-12-02T22:24:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2560092160 bytes 2023-12-02T22:24:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2677070848 bytes 2023-12-02T22:24:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2677666816 bytes 2023-12-02T22:24:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2724433920 bytes 2023-12-02T22:24:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2725035008 bytes 2023-12-02T22:24:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2810033152 bytes 2023-12-02T22:24:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2896870400 bytes 2023-12-02T22:24:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2942052352 bytes 2023-12-02T22:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2994223104 bytes 2023-12-02T22:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3007791104 bytes 2023-12-02T22:24:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3049539584 bytes 2023-12-02T22:24:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3050127360 bytes 2023-12-02T22:24:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3098979328 bytes 2023-12-02T22:24:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3119122432 bytes 2023-12-02T22:24:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3149294592 bytes 2023-12-02T22:24:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3195608064 bytes 2023-12-02T22:24:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3217266688 bytes 2023-12-02T22:24:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3241722880 bytes 2023-12-02T22:24:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3291110400 bytes 2023-12-02T22:24:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3307893760 bytes 2023-12-02T22:24:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3334766592 bytes 2023-12-02T22:24:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3376643072 bytes 2023-12-02T22:24:12.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3437546496 bytes 2023-12-02T22:24:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3454171136 bytes 2023-12-02T22:24:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3492051968 bytes 2023-12-02T22:24:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3523696640 bytes 2023-12-02T22:24:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3568850944 bytes 2023-12-02T22:24:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3600393216 bytes 2023-12-02T22:24:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3601038336 bytes 2023-12-02T22:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3706885120 bytes 2023-12-02T22:24:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 3722571776 bytes 2023-12-02T22:24:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 3744090112 bytes 2023-12-02T22:24:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 3744685056 bytes 2023-12-02T22:24:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 3815667712 bytes 2023-12-02T22:24:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 3886048256 bytes 2023-12-02T22:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 3923419136 bytes 2023-12-02T22:24:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 3926277120 bytes 2023-12-02T22:24:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 3972932608 bytes 2023-12-02T22:24:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4031282176 bytes 2023-12-02T22:24:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4070396928 bytes 2023-12-02T22:24:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4104945664 bytes 2023-12-02T22:24:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:24:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:24:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:24:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:24:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:24:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:24:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:24:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:24:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:24:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":17538901.33,"bytes_per_snapshot":263083520.0,"last_snapshot_bytes":526167040,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555836,"remote_snapshot_timestamp":1701555836,"replay_state":"idle"} 2023-12-02T22:24:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:24:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:24:10 2023-12-02T22:24:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:24:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:24:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:24:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:24:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:24:11 2023-12-02T22:24:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:24:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":17538901.33,"bytes_per_snapshot":263083520.0,"last_snapshot_bytes":526167040,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555836,"remote_snapshot_timestamp":1701555836,"replay_state":"idle"} 2023-12-02T22:24:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:24:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:24:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:24:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:24:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:24:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:24:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:24:10' 2023-12-02T22:24:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:24:10' 2023-12-02T22:24:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:24:10' 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:24:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:24:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:24:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:24:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:24:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:24:10' 2023-12-02T22:24:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:24:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:24:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:24:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:24:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap2 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2023-12-02T22:24:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:24:29.147 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3237951702591876) > weight (0.25) 2023-12-02T22:24:29.147 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-02T22:24:29.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-02T22:24:29.148 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T22:24:29.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:24:29.148+0000 7ffb2c832640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 90467) UID: 1000 2023-12-02T22:24:30.149 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-02T22:24:30.149 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:24:30.149 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-02T22:24:30.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T22:24:30.150 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T22:24:30.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T22:24:30.153 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2023-12-02T22:24:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:24:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:24:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:24:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:24:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:24:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:24:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:24:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:24:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:24:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:24:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:24:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:24:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:24:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:24:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:24:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:24:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:24:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:24:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:24:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:24:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:24:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:24:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:24:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:24:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:24:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:24:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:24:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:24:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:24:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:24:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:24:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:24:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:24:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:24:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:24:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:24:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:24:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:24:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:24:46.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:24:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:24:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:24:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:24:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:24:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:24:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:24:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:24:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:24:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:24:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:24:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:24:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:24:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:24:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:24:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 9 2023-12-02T22:24:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:24:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:24:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:24:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:24:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:24:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=9 2023-12-02T22:24:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:24:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:24:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:24:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:24:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:24:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:24:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:24:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:24:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:24:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:24:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:24:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 9 = '' 2023-12-02T22:24:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:24:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:24:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:24:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:24:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:24:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:24:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:24:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:24:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:24:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:24:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 9 = '' 2023-12-02T22:24:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:24:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:24:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:24:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:24:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:24:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:24:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:24:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:24:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 9 = '' 2023-12-02T22:24:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:24:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:24:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:24:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:24:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=9 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 9 = 9 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:24:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:24:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 4 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:24:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:24:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:24:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:24:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:24:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:24:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap1-clean ']' 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1-clean 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:24:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:24:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:24:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:24:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:24:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:24:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap2-clean 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2-clean 2023-12-02T22:24:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:24:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 30504960 bytes 2023-12-02T22:24:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 72000512 bytes 2023-12-02T22:24:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 108083200 bytes 2023-12-02T22:24:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 120012800 bytes 2023-12-02T22:24:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 180756480 bytes 2023-12-02T22:24:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 222291968 bytes 2023-12-02T22:24:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 277703680 bytes 2023-12-02T22:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 376353792 bytes 2023-12-02T22:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 395760640 bytes 2023-12-02T22:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 481070080 bytes 2023-12-02T22:24:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 488698880 bytes 2023-12-02T22:24:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 532984832 bytes 2023-12-02T22:24:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 564472832 bytes 2023-12-02T22:24:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 659005440 bytes 2023-12-02T22:24:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 668225536 bytes 2023-12-02T22:24:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 745583616 bytes 2023-12-02T22:24:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 809143296 bytes 2023-12-02T22:24:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 857802752 bytes 2023-12-02T22:24:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 887544832 bytes 2023-12-02T22:24:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 900317184 bytes 2023-12-02T22:24:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 901272576 bytes 2023-12-02T22:24:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 969299968 bytes 2023-12-02T22:24:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1022643200 bytes 2023-12-02T22:24:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1040361472 bytes 2023-12-02T22:24:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1091207168 bytes 2023-12-02T22:24:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1142628352 bytes 2023-12-02T22:24:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1193397248 bytes 2023-12-02T22:24:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1243408384 bytes 2023-12-02T22:24:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1274383360 bytes 2023-12-02T22:24:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1324178432 bytes 2023-12-02T22:24:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1352470528 bytes 2023-12-02T22:24:59.382 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:24:59.380+0000 7f2e21bfd640 -1 reset not still connected to 0x557cdfae1c70 2023-12-02T22:24:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1391528960 bytes 2023-12-02T22:24:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1404229632 bytes 2023-12-02T22:24:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1478627328 bytes 2023-12-02T22:24:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1534708736 bytes 2023-12-02T22:24:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1560260608 bytes 2023-12-02T22:24:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1626291200 bytes 2023-12-02T22:24:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1641189376 bytes 2023-12-02T22:25:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1687590912 bytes 2023-12-02T22:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1713581056 bytes 2023-12-02T22:25:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1714209792 bytes 2023-12-02T22:25:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1747581952 bytes 2023-12-02T22:25:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1769121792 bytes 2023-12-02T22:25:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1801423872 bytes 2023-12-02T22:25:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1862259712 bytes 2023-12-02T22:25:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1894303744 bytes 2023-12-02T22:25:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1991199744 bytes 2023-12-02T22:25:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2030798848 bytes 2023-12-02T22:25:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2062282752 bytes 2023-12-02T22:25:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2071446528 bytes 2023-12-02T22:25:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2129258496 bytes 2023-12-02T22:25:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2174878720 bytes 2023-12-02T22:25:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2245340160 bytes 2023-12-02T22:25:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2268627968 bytes 2023-12-02T22:25:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2318965760 bytes 2023-12-02T22:25:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2387499008 bytes 2023-12-02T22:25:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2446479360 bytes 2023-12-02T22:25:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2485848064 bytes 2023-12-02T22:25:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2535445504 bytes 2023-12-02T22:25:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2598272000 bytes 2023-12-02T22:25:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2613437440 bytes 2023-12-02T22:25:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2638490624 bytes 2023-12-02T22:25:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2676481024 bytes 2023-12-02T22:25:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2718320640 bytes 2023-12-02T22:25:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2772806656 bytes 2023-12-02T22:25:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2832714752 bytes 2023-12-02T22:25:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2870568960 bytes 2023-12-02T22:25:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2901737472 bytes 2023-12-02T22:25:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2969731072 bytes 2023-12-02T22:25:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3026557952 bytes 2023-12-02T22:25:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3092422656 bytes 2023-12-02T22:25:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3128007680 bytes 2023-12-02T22:25:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3171011584 bytes 2023-12-02T22:25:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3183256576 bytes 2023-12-02T22:25:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3198895104 bytes 2023-12-02T22:25:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3268596736 bytes 2023-12-02T22:25:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3305339904 bytes 2023-12-02T22:25:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3360579584 bytes 2023-12-02T22:25:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3386984448 bytes 2023-12-02T22:25:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3423403008 bytes 2023-12-02T22:25:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3455116288 bytes 2023-12-02T22:25:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3520709632 bytes 2023-12-02T22:25:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3585209344 bytes 2023-12-02T22:25:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3668958208 bytes 2023-12-02T22:25:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3684655104 bytes 2023-12-02T22:25:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3722287104 bytes 2023-12-02T22:25:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3744844800 bytes 2023-12-02T22:25:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3761302528 bytes 2023-12-02T22:25:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3786478592 bytes 2023-12-02T22:25:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3803375616 bytes 2023-12-02T22:25:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3855490048 bytes 2023-12-02T22:25:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3864070144 bytes 2023-12-02T22:25:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3900554240 bytes 2023-12-02T22:25:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3980061696 bytes 2023-12-02T22:25:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3992020992 bytes 2023-12-02T22:25:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4013129728 bytes 2023-12-02T22:25:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4054336512 bytes 2023-12-02T22:25:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4116116480 bytes 2023-12-02T22:25:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4142508032 bytes 2023-12-02T22:25:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4168509440 bytes 2023-12-02T22:25:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4189419520 bytes 2023-12-02T22:25:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4207371264 bytes 2023-12-02T22:25:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4275008512 bytes 2023-12-02T22:25:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4318543872 bytes 2023-12-02T22:25:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4364338176 bytes 2023-12-02T22:25:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4411083776 bytes 2023-12-02T22:25:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4433303552 bytes 2023-12-02T22:25:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10800 IOs, 4477364224 bytes 2023-12-02T22:25:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10900 IOs, 4525872128 bytes 2023-12-02T22:25:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11000 IOs, 4591261696 bytes 2023-12-02T22:25:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11100 IOs, 4608564224 bytes 2023-12-02T22:25:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11200 IOs, 4636188672 bytes 2023-12-02T22:25:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11300 IOs, 4678754304 bytes 2023-12-02T22:25:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11400 IOs, 4686380032 bytes 2023-12-02T22:25:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11500 IOs, 4713672704 bytes 2023-12-02T22:25:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11600 IOs, 4764552192 bytes 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:25:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:25:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:25:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:25:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":263083520.0,"last_snapshot_bytes":526167040,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555836,"remote_snapshot_timestamp":1701555836,"replay_state":"idle"} 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:24:40 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:25:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:24:41 2023-12-02T22:25:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:25:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":263083520.0,"last_snapshot_bytes":526167040,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555836,"remote_snapshot_timestamp":1701555836,"replay_state":"idle"} 2023-12-02T22:25:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:25:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:25:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:25:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:25:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:25:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:24:40' 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:24:40' 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:24:40' 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:25:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:25:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:25:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:25:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:25:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:25:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:24:40' 2023-12-02T22:25:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:25:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:25:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:25:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:25:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:25:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3 2023-12-02T22:25:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap3 2023-12-02T22:25:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2023-12-02T22:25:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:25:12.152 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T22:25:12.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T22:25:12.153 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-02T22:25:12.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:25:12.153+0000 7f119efcf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 90588) UID: 1000 2023-12-02T22:25:15.154 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T22:25:15.154 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:25:15.155 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T22:25:15.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:25:15.155 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:25:15.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:25:15.158 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-02T22:25:37.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.848595554426323) > weight (0.25) 2023-12-02T22:25:37.154 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-02T22:25:37.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-02T22:25:37.154 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-02T22:25:37.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:25:37.154+0000 7f401a8e6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 92776) UID: 1000 2023-12-02T22:25:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:25:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:25:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:25:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:25:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:25:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:25:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:25:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:25:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:25:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:25:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:25:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:25:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:25:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:25:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:25:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:25:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:25:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:25:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:25:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:25:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:25:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:25:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:25:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:25:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:25:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:25:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:25:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:25:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:25:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:25:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:25:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 13 2023-12-02T22:25:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:25:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:25:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:25:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:25:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=13 2023-12-02T22:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:25:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:25:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:25:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:25:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:25:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:25:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:25:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:25:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:25:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 13 = '' 2023-12-02T22:25:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:25:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:25:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:25:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:25:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:25:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:25:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:25:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:25:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:25:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 13 = '' 2023-12-02T22:25:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:25:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:25:43.155 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-02T22:25:43.155 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:25:43.155 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-02T22:25:43.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T22:25:43.156 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T22:25:43.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T22:25:43.159 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:25:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:25:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:25:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:25:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:25:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 13 = '' 2023-12-02T22:25:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:25:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:25:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:25:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:25:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:25:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=13 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 13 = 13 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:25:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 6 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:25:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:24:52 2023 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:25:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap2-clean ']' 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2-clean 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:25:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:25:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:25:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:25:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:25:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:25:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:25:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:25:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:25:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap3-clean 2023-12-02T22:25:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-02T22:25:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3-clean 2023-12-02T22:25:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:25:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:25:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:25:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:25:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 27348992 bytes 2023-12-02T22:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 27970560 bytes 2023-12-02T22:25:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 98966528 bytes 2023-12-02T22:25:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 99581952 bytes 2023-12-02T22:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 113457152 bytes 2023-12-02T22:25:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 157517824 bytes 2023-12-02T22:25:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 190845952 bytes 2023-12-02T22:25:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 251902976 bytes 2023-12-02T22:25:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 298485760 bytes 2023-12-02T22:25:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 339074048 bytes 2023-12-02T22:25:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 354811904 bytes 2023-12-02T22:25:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 374589440 bytes 2023-12-02T22:25:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 376243200 bytes 2023-12-02T22:25:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 396054528 bytes 2023-12-02T22:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 430385152 bytes 2023-12-02T22:25:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 431077376 bytes 2023-12-02T22:25:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 520111104 bytes 2023-12-02T22:25:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 546410496 bytes 2023-12-02T22:25:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 615668736 bytes 2023-12-02T22:25:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 732846080 bytes 2023-12-02T22:25:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 766176256 bytes 2023-12-02T22:25:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 827666432 bytes 2023-12-02T22:25:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 857518080 bytes 2023-12-02T22:25:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 891090944 bytes 2023-12-02T22:25:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 980066304 bytes 2023-12-02T22:25:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 995115008 bytes 2023-12-02T22:25:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1063097344 bytes 2023-12-02T22:25:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1100901376 bytes 2023-12-02T22:25:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1153613824 bytes 2023-12-02T22:25:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1197629440 bytes 2023-12-02T22:25:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1270425600 bytes 2023-12-02T22:25:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1330214912 bytes 2023-12-02T22:25:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1383363584 bytes 2023-12-02T22:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1395454976 bytes 2023-12-02T22:25:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1407394816 bytes 2023-12-02T22:25:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1463216128 bytes 2023-12-02T22:25:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1508268032 bytes 2023-12-02T22:25:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1565783040 bytes 2023-12-02T22:25:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1619318784 bytes 2023-12-02T22:25:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1698577408 bytes 2023-12-02T22:25:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1726213120 bytes 2023-12-02T22:25:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1787485184 bytes 2023-12-02T22:25:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1858174976 bytes 2023-12-02T22:25:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1925762048 bytes 2023-12-02T22:25:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1977613312 bytes 2023-12-02T22:25:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2013970432 bytes 2023-12-02T22:25:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2038316032 bytes 2023-12-02T22:25:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2100698112 bytes 2023-12-02T22:25:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2138079232 bytes 2023-12-02T22:25:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2160996352 bytes 2023-12-02T22:25:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2179556352 bytes 2023-12-02T22:25:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2211558400 bytes 2023-12-02T22:25:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2293782528 bytes 2023-12-02T22:25:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2317950976 bytes 2023-12-02T22:25:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2333646848 bytes 2023-12-02T22:25:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2338493440 bytes 2023-12-02T22:25:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:25:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:25:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:25:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:25:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:25:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:25:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:25:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:25:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:25:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:25:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:25:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530616320.0,"last_snapshot_bytes":535065600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555886,"remote_snapshot_timestamp":1701555886,"replay_state":"idle"} 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:25:40 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:25:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:25:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:25:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:25:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:25:44 2023-12-02T22:25:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:25:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530616320.0,"last_snapshot_bytes":535065600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555886,"remote_snapshot_timestamp":1701555886,"replay_state":"idle"} 2023-12-02T22:25:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:25:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:25:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:25:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:25:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:25:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:25:40' 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:25:40' 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:25:40' 2023-12-02T22:25:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:25:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:25:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:25:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:25:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:25:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:25:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:25:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:25:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:25:40' 2023-12-02T22:25:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:25:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:25:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:25:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:25:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap4 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2023-12-02T22:25:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:26:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:26:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:26:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:26:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:26:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:26:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:26:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:26:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:26:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:26:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:26:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:26:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:26:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:26:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:26:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:26:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:26:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:26:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:26:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:26:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:26:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:26:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:26:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:26:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:26:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 17 2023-12-02T22:26:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:26:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:26:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:26:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:26:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:26:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=17 2023-12-02T22:26:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:26:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:26:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:26:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:26:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:26:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:26:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:26:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:26:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:26:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:26:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 17 = '' 2023-12-02T22:26:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:26:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:26:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:26:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:26:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:26:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:26:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:26:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:26:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:26:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 17 = '' 2023-12-02T22:26:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:26:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:26:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:26:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:26:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:26:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:26:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:26:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:26:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:26:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 17 = '' 2023-12-02T22:26:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:26:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:26:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:26:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=17 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 17 = 17 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:26:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-02T22:26:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:26:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:26:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 8 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:24:52 2023 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:26:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:26:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:26:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:26:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:26:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:26:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap3-clean ']' 2023-12-02T22:26:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3-clean 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:26:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:26:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:26:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:26:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:26:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:26:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap4-clean 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4-clean 2023-12-02T22:26:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:26:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:26:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:26:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:26:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:26:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:26:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:26:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 48519168 bytes 2023-12-02T22:26:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 119286784 bytes 2023-12-02T22:26:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 162000896 bytes 2023-12-02T22:26:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 205167616 bytes 2023-12-02T22:26:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 248162304 bytes 2023-12-02T22:26:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 265341952 bytes 2023-12-02T22:26:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 302776320 bytes 2023-12-02T22:26:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 344825856 bytes 2023-12-02T22:26:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 381852672 bytes 2023-12-02T22:26:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 404408320 bytes 2023-12-02T22:26:40.159 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.745985751053199) > weight (0.25) 2023-12-02T22:26:40.159 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T22:26:40.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T22:26:40.160 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T22:26:40.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:26:40.160+0000 7f888545c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 90650) UID: 1000 2023-12-02T22:26:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 474497024 bytes 2023-12-02T22:26:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 532149248 bytes 2023-12-02T22:26:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 569799680 bytes 2023-12-02T22:26:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 570415104 bytes 2023-12-02T22:26:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 571053056 bytes 2023-12-02T22:26:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 616862720 bytes 2023-12-02T22:26:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 641474560 bytes 2023-12-02T22:26:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 661553152 bytes 2023-12-02T22:26:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 719180800 bytes 2023-12-02T22:26:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 802604032 bytes 2023-12-02T22:26:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 880814080 bytes 2023-12-02T22:26:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 932200448 bytes 2023-12-02T22:26:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1001717760 bytes 2023-12-02T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1063630848 bytes 2023-12-02T22:26:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1174757376 bytes 2023-12-02T22:26:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1213456384 bytes 2023-12-02T22:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1255862272 bytes 2023-12-02T22:26:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1324323840 bytes 2023-12-02T22:26:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1340732416 bytes 2023-12-02T22:26:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1360651264 bytes 2023-12-02T22:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1361261568 bytes 2023-12-02T22:26:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1361866752 bytes 2023-12-02T22:26:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1390329856 bytes 2023-12-02T22:26:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1464169472 bytes 2023-12-02T22:26:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1538203648 bytes 2023-12-02T22:26:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1600882688 bytes 2023-12-02T22:26:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1654721536 bytes 2023-12-02T22:26:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1681523712 bytes 2023-12-02T22:26:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1739324416 bytes 2023-12-02T22:26:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1771047936 bytes 2023-12-02T22:26:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1822977024 bytes 2023-12-02T22:26:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1851079680 bytes 2023-12-02T22:26:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1919017984 bytes 2023-12-02T22:26:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1938718720 bytes 2023-12-02T22:26:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1972592640 bytes 2023-12-02T22:26:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2010580992 bytes 2023-12-02T22:26:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2166195200 bytes 2023-12-02T22:26:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2181368832 bytes 2023-12-02T22:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2211261440 bytes 2023-12-02T22:26:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2262507520 bytes 2023-12-02T22:26:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2312352768 bytes 2023-12-02T22:26:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2355640320 bytes 2023-12-02T22:26:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2436030464 bytes 2023-12-02T22:26:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2499289088 bytes 2023-12-02T22:26:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2555564032 bytes 2023-12-02T22:26:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2617405440 bytes 2023-12-02T22:26:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2618009600 bytes 2023-12-02T22:26:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2637545472 bytes 2023-12-02T22:26:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2671152128 bytes 2023-12-02T22:26:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2671758336 bytes 2023-12-02T22:26:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2728867840 bytes 2023-12-02T22:26:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2779940864 bytes 2023-12-02T22:26:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2798434304 bytes 2023-12-02T22:26:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2880883712 bytes 2023-12-02T22:26:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2918635520 bytes 2023-12-02T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2973703168 bytes 2023-12-02T22:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3021394944 bytes 2023-12-02T22:26:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3072733184 bytes 2023-12-02T22:26:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3143717888 bytes 2023-12-02T22:26:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3185801216 bytes 2023-12-02T22:26:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3278512128 bytes 2023-12-02T22:26:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3322276864 bytes 2023-12-02T22:26:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3379087360 bytes 2023-12-02T22:26:47.161 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T22:26:47.161 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:26:47.162 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T22:26:47.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T22:26:47.162 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T22:26:47.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T22:26:47.165 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-12-02T22:26:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3437728768 bytes 2023-12-02T22:26:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3489812480 bytes 2023-12-02T22:26:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3518503936 bytes 2023-12-02T22:26:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3519584256 bytes 2023-12-02T22:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3536919552 bytes 2023-12-02T22:26:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3556489216 bytes 2023-12-02T22:26:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3623720960 bytes 2023-12-02T22:26:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3683960832 bytes 2023-12-02T22:26:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3713042432 bytes 2023-12-02T22:26:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3774729216 bytes 2023-12-02T22:26:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3817446400 bytes 2023-12-02T22:26:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3840541696 bytes 2023-12-02T22:26:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3878437888 bytes 2023-12-02T22:26:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3925514240 bytes 2023-12-02T22:26:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3961480192 bytes 2023-12-02T22:26:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3968406528 bytes 2023-12-02T22:26:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3983184896 bytes 2023-12-02T22:26:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 4028922880 bytes 2023-12-02T22:26:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 4045713408 bytes 2023-12-02T22:26:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 4082038784 bytes 2023-12-02T22:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 4082644992 bytes 2023-12-02T22:26:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 4131017728 bytes 2023-12-02T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4216617984 bytes 2023-12-02T22:26:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4217199616 bytes 2023-12-02T22:26:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4232786944 bytes 2023-12-02T22:26:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4262755328 bytes 2023-12-02T22:26:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4304882688 bytes 2023-12-02T22:26:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4311469056 bytes 2023-12-02T22:26:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4390455296 bytes 2023-12-02T22:26:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4435315712 bytes 2023-12-02T22:26:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4495968256 bytes 2023-12-02T22:26:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4599428096 bytes 2023-12-02T22:26:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4627251200 bytes 2023-12-02T22:26:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4688917504 bytes 2023-12-02T22:26:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10800 IOs, 4728268800 bytes 2023-12-02T22:26:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10900 IOs, 4778470400 bytes 2023-12-02T22:26:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:26:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:26:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:26:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:26:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:26:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:26:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:26:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":17690487.47,"bytes_per_snapshot":533765632.0,"last_snapshot_bytes":530714624,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555989,"remote_snapshot_timestamp":1701555989,"replay_state":"idle"} 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:26:40 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: down+unknown 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: status not found 2023-12-02T22:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02T22:26:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:26:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":17690487.47,"bytes_per_snapshot":533765632.0,"last_snapshot_bytes":530714624,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701555989,"remote_snapshot_timestamp":1701555989,"replay_state":"idle"} 2023-12-02T22:26:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:26:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:26:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:26:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:26:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:26:40' 2023-12-02T22:26:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:26:40' 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:26:40' 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:26:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:26:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:26:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:26:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:26:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:26:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:26:40' 2023-12-02T22:26:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:26:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:26:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:26:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:26:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap5 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:26:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2023-12-02T22:26:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:27:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:27:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:27:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:27:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:27:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:27:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:27:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:27:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:27:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:27:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:27:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:27:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:27:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:27:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:27:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:27:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:27:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:27:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:27:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:27:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:27:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:27:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:27:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:27:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:27:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:27:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:27:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:27:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:27:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:27:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:27:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:27:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:27:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:27:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:27:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:27:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 21 2023-12-02T22:27:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:27:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:27:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:27:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:27:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:27:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=21 2023-12-02T22:27:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:27:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:27:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:27:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:27:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 21 = '' 2023-12-02T22:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:27:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:27:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:27:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:27:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:27:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:27:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:27:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:27:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:27:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 21 = '' 2023-12-02T22:27:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:27:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:27:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:27:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:27:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:27:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:27:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:27:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 21 = '' 2023-12-02T22:27:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:27:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:27:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:27:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:27:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:27:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:27:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=21 2023-12-02T22:27:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 21 = 21 2023-12-02T22:27:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-02T22:27:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2023-12-02T22:27:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:27:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:27:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:27:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 10 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:27:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:26:35 2023 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:27:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap4-clean ']' 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4-clean 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:27:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:27:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:27:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:27:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:27:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:27:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:27:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:27:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:27:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap5-clean 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5-clean 2023-12-02T22:27:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:27:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:27:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:27:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:27:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:27:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:27:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:27:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 79397888 bytes 2023-12-02T22:27:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 156898304 bytes 2023-12-02T22:27:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 170256384 bytes 2023-12-02T22:27:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 216609792 bytes 2023-12-02T22:27:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 298229760 bytes 2023-12-02T22:27:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 347952128 bytes 2023-12-02T22:27:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 440600576 bytes 2023-12-02T22:27:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 486011904 bytes 2023-12-02T22:27:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 532134912 bytes 2023-12-02T22:27:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 589161472 bytes 2023-12-02T22:27:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 654260224 bytes 2023-12-02T22:27:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 699016192 bytes 2023-12-02T22:27:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 719062016 bytes 2023-12-02T22:27:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 734772224 bytes 2023-12-02T22:27:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 767156224 bytes 2023-12-02T22:27:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 807019520 bytes 2023-12-02T22:27:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 861779968 bytes 2023-12-02T22:27:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 899578880 bytes 2023-12-02T22:27:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 921768960 bytes 2023-12-02T22:27:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 954450944 bytes 2023-12-02T22:27:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 985874432 bytes 2023-12-02T22:27:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1055963136 bytes 2023-12-02T22:27:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1066137600 bytes 2023-12-02T22:27:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1101485056 bytes 2023-12-02T22:27:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1120727040 bytes 2023-12-02T22:27:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1149513728 bytes 2023-12-02T22:27:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1182401536 bytes 2023-12-02T22:27:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1257059328 bytes 2023-12-02T22:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1280159744 bytes 2023-12-02T22:27:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1329382400 bytes 2023-12-02T22:27:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1338506240 bytes 2023-12-02T22:27:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1363762176 bytes 2023-12-02T22:27:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1419974656 bytes 2023-12-02T22:27:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1449944064 bytes 2023-12-02T22:27:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1485360128 bytes 2023-12-02T22:27:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1545913344 bytes 2023-12-02T22:27:38.159 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4658735826382774) > weight (0.25) 2023-12-02T22:27:38.159 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5630414683511394) > weight (0.25) 2023-12-02T22:27:38.159 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6973559614545471) > weight (0.25) 2023-12-02T22:27:38.159 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T22:27:38.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T22:27:38.160 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-02T22:27:38.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:27:38.159+0000 7f9925e43640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 90548) UID: 1000 2023-12-02T22:27:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1624337408 bytes 2023-12-02T22:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1753614336 bytes 2023-12-02T22:27:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1802034176 bytes 2023-12-02T22:27:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1834999808 bytes 2023-12-02T22:27:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1845755904 bytes 2023-12-02T22:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1865248768 bytes 2023-12-02T22:27:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1900780544 bytes 2023-12-02T22:27:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1976331264 bytes 2023-12-02T22:27:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2036703232 bytes 2023-12-02T22:27:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2069131264 bytes 2023-12-02T22:27:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2074547200 bytes 2023-12-02T22:27:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2112238592 bytes 2023-12-02T22:27:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2132442112 bytes 2023-12-02T22:27:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2157987840 bytes 2023-12-02T22:27:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2189622272 bytes 2023-12-02T22:27:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2215858176 bytes 2023-12-02T22:27:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2266847232 bytes 2023-12-02T22:27:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2313063424 bytes 2023-12-02T22:27:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2335496192 bytes 2023-12-02T22:27:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2375266304 bytes 2023-12-02T22:27:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2436078592 bytes 2023-12-02T22:27:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2479009792 bytes 2023-12-02T22:27:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2501666816 bytes 2023-12-02T22:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2510539776 bytes 2023-12-02T22:27:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2547767296 bytes 2023-12-02T22:27:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2623412224 bytes 2023-12-02T22:27:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2631315456 bytes 2023-12-02T22:27:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2676664320 bytes 2023-12-02T22:27:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2762407936 bytes 2023-12-02T22:27:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2805361664 bytes 2023-12-02T22:27:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2834850816 bytes 2023-12-02T22:27:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2844612608 bytes 2023-12-02T22:27:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2888503296 bytes 2023-12-02T22:27:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2928436224 bytes 2023-12-02T22:27:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2936514560 bytes 2023-12-02T22:27:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2954454016 bytes 2023-12-02T22:27:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2956136448 bytes 2023-12-02T22:27:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3026484224 bytes 2023-12-02T22:27:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3105610752 bytes 2023-12-02T22:27:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3199127552 bytes 2023-12-02T22:27:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3238137856 bytes 2023-12-02T22:27:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3244589056 bytes 2023-12-02T22:27:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3254168576 bytes 2023-12-02T22:27:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3323381760 bytes 2023-12-02T22:27:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3324004352 bytes 2023-12-02T22:27:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3339038720 bytes 2023-12-02T22:27:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3398924288 bytes 2023-12-02T22:27:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3444785152 bytes 2023-12-02T22:27:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3454358528 bytes 2023-12-02T22:27:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3492898816 bytes 2023-12-02T22:27:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3601283072 bytes 2023-12-02T22:27:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3643846656 bytes 2023-12-02T22:27:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3676905472 bytes 2023-12-02T22:27:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3737852928 bytes 2023-12-02T22:27:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3815696384 bytes 2023-12-02T22:27:44.161 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T22:27:44.161 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:27:44.161 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T22:27:44.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T22:27:44.161 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T22:27:44.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T22:27:44.166 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-02T22:27:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3886396416 bytes 2023-12-02T22:27:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3930306560 bytes 2023-12-02T22:27:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3968841728 bytes 2023-12-02T22:27:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3985959936 bytes 2023-12-02T22:27:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4002734080 bytes 2023-12-02T22:27:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4086789120 bytes 2023-12-02T22:27:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4145480704 bytes 2023-12-02T22:27:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4183362560 bytes 2023-12-02T22:27:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4281230336 bytes 2023-12-02T22:27:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4312552448 bytes 2023-12-02T22:27:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4318341120 bytes 2023-12-02T22:27:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4354549760 bytes 2023-12-02T22:27:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4355158016 bytes 2023-12-02T22:27:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4411924480 bytes 2023-12-02T22:27:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4438365184 bytes 2023-12-02T22:27:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4481979392 bytes 2023-12-02T22:27:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:27:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:27:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:27:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:27:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:27:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:27:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:27:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:27:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":17606041.6,"bytes_per_snapshot":529447936.0,"last_snapshot_bytes":528181248,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556043,"remote_snapshot_timestamp":1701556043,"replay_state":"idle"} 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:27:40 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:27:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:27:41 2023-12-02T22:27:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:27:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":17606041.6,"bytes_per_snapshot":529447936.0,"last_snapshot_bytes":528181248,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556043,"remote_snapshot_timestamp":1701556043,"replay_state":"idle"} 2023-12-02T22:27:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:27:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:27:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:27:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:27:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:27:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:27:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:27:40' 2023-12-02T22:27:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:27:40' 2023-12-02T22:27:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:27:40' 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:27:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:27:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:27:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:27:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:27:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:27:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:27:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:27:40' 2023-12-02T22:27:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:27:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:27:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:27:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:27:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap6 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2023-12-02T22:27:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:28:06.165 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5675201400770089) > weight (0.25) 2023-12-02T22:28:06.166 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4280994172065996) > weight (0.25) 2023-12-02T22:28:06.166 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7294739154127058) > weight (0.25) 2023-12-02T22:28:06.166 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7374933496341144) > weight (0.25) 2023-12-02T22:28:06.166 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-02T22:28:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:28:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:28:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:28:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:28:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:28:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:28:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:28:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:28:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:28:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:28:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:28:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:28:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:28:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:28:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:28:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:28:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:28:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:28:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:28:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:28:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:28:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:28:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:28:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:28:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:28:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:28:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:28:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:28:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:28:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 25 2023-12-02T22:28:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:28:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:28:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:28:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:28:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:28:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=25 2023-12-02T22:28:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:28:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:28:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:28:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:28:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:28:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:28:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:28:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:28:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:28:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:28:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:28:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 25 = '' 2023-12-02T22:28:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:28:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:28:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:28:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:28:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:28:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:28:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:28:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:28:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 25 = '' 2023-12-02T22:28:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:28:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:28:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:28:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:28:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:28:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:28:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:28:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:28:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:28:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 25 = '' 2023-12-02T22:28:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:28:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:28:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:28:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=25 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 25 = 25 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:28:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:28:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 12 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:28:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:26:35 2023 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:28:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:28:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap5-clean ']' 2023-12-02T22:28:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-02T22:28:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5-clean 2023-12-02T22:28:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:28:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:28:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:28:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:28:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:28:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:28:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:28:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:28:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:28:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap6-clean 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6-clean 2023-12-02T22:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:28:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:28:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:28:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:28:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:28:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:28:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:28:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 15221760 bytes 2023-12-02T22:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 35490816 bytes 2023-12-02T22:28:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 75000832 bytes 2023-12-02T22:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 150848512 bytes 2023-12-02T22:28:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 163530752 bytes 2023-12-02T22:28:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 222540800 bytes 2023-12-02T22:28:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 268012544 bytes 2023-12-02T22:28:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 280112128 bytes 2023-12-02T22:28:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 358603776 bytes 2023-12-02T22:28:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 384663552 bytes 2023-12-02T22:28:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 386713600 bytes 2023-12-02T22:28:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 432286720 bytes 2023-12-02T22:28:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 480033792 bytes 2023-12-02T22:28:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 538559488 bytes 2023-12-02T22:28:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 573421568 bytes 2023-12-02T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 677980160 bytes 2023-12-02T22:28:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 720258048 bytes 2023-12-02T22:28:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 765751296 bytes 2023-12-02T22:28:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 815622144 bytes 2023-12-02T22:28:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 884082688 bytes 2023-12-02T22:28:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 963229696 bytes 2023-12-02T22:28:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 989973504 bytes 2023-12-02T22:28:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1067711488 bytes 2023-12-02T22:28:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1131548672 bytes 2023-12-02T22:28:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1241651200 bytes 2023-12-02T22:28:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1242266624 bytes 2023-12-02T22:28:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1242891264 bytes 2023-12-02T22:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1268096000 bytes 2023-12-02T22:28:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1359672320 bytes 2023-12-02T22:28:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1412027392 bytes 2023-12-02T22:28:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1506712576 bytes 2023-12-02T22:28:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1510793216 bytes 2023-12-02T22:28:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1535676416 bytes 2023-12-02T22:28:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1573584896 bytes 2023-12-02T22:28:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1647368192 bytes 2023-12-02T22:28:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1724278784 bytes 2023-12-02T22:28:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1791148032 bytes 2023-12-02T22:28:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1807552512 bytes 2023-12-02T22:28:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1848480768 bytes 2023-12-02T22:28:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1921133568 bytes 2023-12-02T22:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1994068992 bytes 2023-12-02T22:28:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 2035098624 bytes 2023-12-02T22:28:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2076937216 bytes 2023-12-02T22:28:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2082656256 bytes 2023-12-02T22:28:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2083270656 bytes 2023-12-02T22:28:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2104170496 bytes 2023-12-02T22:28:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2123168768 bytes 2023-12-02T22:28:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2147492864 bytes 2023-12-02T22:28:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2208248832 bytes 2023-12-02T22:28:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2251657216 bytes 2023-12-02T22:28:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2330995712 bytes 2023-12-02T22:28:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2356538368 bytes 2023-12-02T22:28:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2418949120 bytes 2023-12-02T22:28:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2492537856 bytes 2023-12-02T22:28:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2520865792 bytes 2023-12-02T22:28:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2540378112 bytes 2023-12-02T22:28:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2550808576 bytes 2023-12-02T22:28:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2618440704 bytes 2023-12-02T22:28:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2693957632 bytes 2023-12-02T22:28:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2744588288 bytes 2023-12-02T22:28:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2822354944 bytes 2023-12-02T22:28:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2894217216 bytes 2023-12-02T22:28:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2909530112 bytes 2023-12-02T22:28:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2950671360 bytes 2023-12-02T22:28:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 3028567040 bytes 2023-12-02T22:28:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 3068847104 bytes 2023-12-02T22:28:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:28:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:28:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:28:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:28:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:28:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:28:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:28:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":529447936.0,"last_snapshot_bytes":528181248,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556043,"remote_snapshot_timestamp":1701556043,"replay_state":"idle"} 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:28:10 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:28:11 2023-12-02T22:28:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:28:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":529447936.0,"last_snapshot_bytes":528181248,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556043,"remote_snapshot_timestamp":1701556043,"replay_state":"idle"} 2023-12-02T22:28:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:28:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:28:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:28:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:28:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:28:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:28:10' 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:28:10' 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:28:10' 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:28:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:28:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:28:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:28:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:28:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:28:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:28:10' 2023-12-02T22:28:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:28:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:28:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:28:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap7 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:28:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:28:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2023-12-02T22:28:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:29:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:29:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:29:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:29:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:29:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:29:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:29:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:29:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:29:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:29:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:29:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:29:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:29:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:29:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:29:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:29:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:29:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:29:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:29:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:29:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:29:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:29:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:29:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:29:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:29:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:29:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:29:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:29:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:29:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:29:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 29 2023-12-02T22:29:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:29:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:29:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:29:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:29:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=29 2023-12-02T22:29:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:29:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:29:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:29:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:29:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:29:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:29:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:29:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 29 = '' 2023-12-02T22:29:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:29:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:29:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:29:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:29:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:29:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:29:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:29:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 29 = '' 2023-12-02T22:29:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:29:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:29:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:29:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:29:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:29:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:29:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 29 = '' 2023-12-02T22:29:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:29:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:29:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=29 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 29 = 29 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:29:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 14 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:29:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:28:24 2023 2023-12-02T22:29:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:29:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:29:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:29:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:29:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:29:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap6-clean ']' 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6-clean 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:29:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:29:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:29:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:29:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:29:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:29:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:29:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap7-clean 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7-clean 2023-12-02T22:29:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:29:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:29:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:29:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:29:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:29:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:29:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:29:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 63481856 bytes 2023-12-02T22:29:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 64089088 bytes 2023-12-02T22:29:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 81623040 bytes 2023-12-02T22:29:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 110289920 bytes 2023-12-02T22:29:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 153401344 bytes 2023-12-02T22:29:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 218348544 bytes 2023-12-02T22:29:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 272557056 bytes 2023-12-02T22:29:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 307737600 bytes 2023-12-02T22:29:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 347118592 bytes 2023-12-02T22:29:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 388914176 bytes 2023-12-02T22:29:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 432432128 bytes 2023-12-02T22:29:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 484263936 bytes 2023-12-02T22:29:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 514215936 bytes 2023-12-02T22:29:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 545935360 bytes 2023-12-02T22:29:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 578552832 bytes 2023-12-02T22:29:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 591911936 bytes 2023-12-02T22:29:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 624081920 bytes 2023-12-02T22:29:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 645877760 bytes 2023-12-02T22:29:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 674924544 bytes 2023-12-02T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 758403072 bytes 2023-12-02T22:29:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 820667392 bytes 2023-12-02T22:29:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 822839296 bytes 2023-12-02T22:29:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 875814912 bytes 2023-12-02T22:29:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 887367680 bytes 2023-12-02T22:29:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 911823872 bytes 2023-12-02T22:29:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 987547648 bytes 2023-12-02T22:29:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1034906624 bytes 2023-12-02T22:29:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1058488320 bytes 2023-12-02T22:29:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1081846784 bytes 2023-12-02T22:29:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1094117376 bytes 2023-12-02T22:29:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1126480896 bytes 2023-12-02T22:29:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1165908992 bytes 2023-12-02T22:29:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1188284416 bytes 2023-12-02T22:29:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1222585344 bytes 2023-12-02T22:29:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1257444352 bytes 2023-12-02T22:29:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1307489280 bytes 2023-12-02T22:29:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1330063360 bytes 2023-12-02T22:29:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1352943616 bytes 2023-12-02T22:29:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1390996480 bytes 2023-12-02T22:29:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1479109632 bytes 2023-12-02T22:29:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1487830016 bytes 2023-12-02T22:29:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1488452608 bytes 2023-12-02T22:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1571747840 bytes 2023-12-02T22:29:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1619584000 bytes 2023-12-02T22:29:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1646781440 bytes 2023-12-02T22:29:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1661526016 bytes 2023-12-02T22:29:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1728273408 bytes 2023-12-02T22:29:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1824857088 bytes 2023-12-02T22:29:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1843022848 bytes 2023-12-02T22:29:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1892725760 bytes 2023-12-02T22:29:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1976628224 bytes 2023-12-02T22:29:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2008735744 bytes 2023-12-02T22:29:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2030096384 bytes 2023-12-02T22:29:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2043100160 bytes 2023-12-02T22:29:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2136625152 bytes 2023-12-02T22:29:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2158056448 bytes 2023-12-02T22:29:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2276034560 bytes 2023-12-02T22:29:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2277423104 bytes 2023-12-02T22:29:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2326752256 bytes 2023-12-02T22:29:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2368924672 bytes 2023-12-02T22:29:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2414325760 bytes 2023-12-02T22:29:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2444476416 bytes 2023-12-02T22:29:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2494516224 bytes 2023-12-02T22:29:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2544454656 bytes 2023-12-02T22:29:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2586319872 bytes 2023-12-02T22:29:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2642508800 bytes 2023-12-02T22:29:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2702554112 bytes 2023-12-02T22:29:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2725650432 bytes 2023-12-02T22:29:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2800174080 bytes 2023-12-02T22:29:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:29:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:29:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:29:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:29:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:29:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:29:26.166 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2895888403746202) > weight (0.25) 2023-12-02T22:29:26.167 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.847415850756064) > weight (0.25) 2023-12-02T22:29:26.167 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5558801966561226) > weight (0.25) 2023-12-02T22:29:26.167 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.31810721833969513) > weight (0.25) 2023-12-02T22:29:26.167 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-12-02T22:29:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:29:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:29:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:29:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532043776.0,"last_snapshot_bytes":535906304,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556098,"remote_snapshot_timestamp":1701556098,"replay_state":"idle"} 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:29:10 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:29:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:29:11 2023-12-02T22:29:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:29:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532043776.0,"last_snapshot_bytes":535906304,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556098,"remote_snapshot_timestamp":1701556098,"replay_state":"idle"} 2023-12-02T22:29:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:29:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:29:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:29:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:29:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:29:10' 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:29:10' 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:29:10' 2023-12-02T22:29:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:29:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:29:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:29:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:29:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:29:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:29:10' 2023-12-02T22:29:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:29:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:29:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:29:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:29:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap8 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2023-12-02T22:29:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:29:48.167 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.34367322748327567) > weight (0.25) 2023-12-02T22:29:48.167 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T22:29:48.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T22:29:48.168 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T22:29:48.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:29:48.167+0000 7ff441b61640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 94173) UID: 1000 2023-12-02T22:29:49.168 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T22:29:49.169 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:29:49.169 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T22:29:49.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T22:29:49.169 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T22:29:49.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T22:29:49.172 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-12-02T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:29:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:29:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:29:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:29:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:29:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:29:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:29:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:29:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:29:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:29:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:29:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:29:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:29:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:29:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:29:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:29:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:29:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:29:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:29:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:29:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:29:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:29:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:29:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:29:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:29:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:29:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:29:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:29:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:29:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:29:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:29:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:29:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:29:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:29:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:29:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:29:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:29:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:29:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:29:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 33 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:29:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:29:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=33 2023-12-02T22:29:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:29:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:29:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:29:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:29:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:29:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:29:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:29:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 33 = '' 2023-12-02T22:29:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:29:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:29:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:29:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:29:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:29:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:29:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:29:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 33 = '' 2023-12-02T22:29:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:29:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:30:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:30:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:30:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:30:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:30:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:30:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:30:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:30:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 33 = '' 2023-12-02T22:30:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:30:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:30:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:30:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=33 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 33 = 33 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:30:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:30:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-02T22:30:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:30:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 16 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:28:24 2023 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:30:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:30:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:30:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:30:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap7-clean ']' 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7-clean 2023-12-02T22:30:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:30:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8 2023-12-02T22:30:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2023-12-02T22:30:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:30:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap8-clean 2023-12-02T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-02T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8-clean 2023-12-02T22:30:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:30:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:30:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:30:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:30:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:30:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:30:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:30:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 33662976 bytes 2023-12-02T22:30:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 79960064 bytes 2023-12-02T22:30:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 97152000 bytes 2023-12-02T22:30:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 143448064 bytes 2023-12-02T22:30:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 144087040 bytes 2023-12-02T22:30:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 196515840 bytes 2023-12-02T22:30:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 324263936 bytes 2023-12-02T22:30:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 354576384 bytes 2023-12-02T22:30:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 402251776 bytes 2023-12-02T22:30:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 423199744 bytes 2023-12-02T22:30:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 443901952 bytes 2023-12-02T22:30:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 512056320 bytes 2023-12-02T22:30:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 568589312 bytes 2023-12-02T22:30:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 569225216 bytes 2023-12-02T22:30:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 638490624 bytes 2023-12-02T22:30:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 650449920 bytes 2023-12-02T22:30:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 712361984 bytes 2023-12-02T22:30:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 728920064 bytes 2023-12-02T22:30:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 779603968 bytes 2023-12-02T22:30:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 826483712 bytes 2023-12-02T22:30:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 850802688 bytes 2023-12-02T22:30:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 871074816 bytes 2023-12-02T22:30:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 933581824 bytes 2023-12-02T22:30:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 936504320 bytes 2023-12-02T22:30:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 957497344 bytes 2023-12-02T22:30:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 971155456 bytes 2023-12-02T22:30:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1025125376 bytes 2023-12-02T22:30:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1071540224 bytes 2023-12-02T22:30:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1098776576 bytes 2023-12-02T22:30:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1150103552 bytes 2023-12-02T22:30:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1178813440 bytes 2023-12-02T22:30:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1217806336 bytes 2023-12-02T22:30:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1239717888 bytes 2023-12-02T22:30:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1286595584 bytes 2023-12-02T22:30:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1295379456 bytes 2023-12-02T22:30:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1308249088 bytes 2023-12-02T22:30:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1365738496 bytes 2023-12-02T22:30:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1408206848 bytes 2023-12-02T22:30:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1422848000 bytes 2023-12-02T22:30:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1444133888 bytes 2023-12-02T22:30:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1464369152 bytes 2023-12-02T22:30:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1488238592 bytes 2023-12-02T22:30:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1512107008 bytes 2023-12-02T22:30:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1576866816 bytes 2023-12-02T22:30:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1657402368 bytes 2023-12-02T22:30:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1715998720 bytes 2023-12-02T22:30:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1723480064 bytes 2023-12-02T22:30:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1748201472 bytes 2023-12-02T22:30:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1863161856 bytes 2023-12-02T22:30:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1877017600 bytes 2023-12-02T22:30:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1905473536 bytes 2023-12-02T22:30:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1996224512 bytes 2023-12-02T22:30:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2062164992 bytes 2023-12-02T22:30:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2105497600 bytes 2023-12-02T22:30:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2137264128 bytes 2023-12-02T22:30:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2171641856 bytes 2023-12-02T22:30:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2200906752 bytes 2023-12-02T22:30:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2263912448 bytes 2023-12-02T22:30:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2286989312 bytes 2023-12-02T22:30:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2324749312 bytes 2023-12-02T22:30:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2353814528 bytes 2023-12-02T22:30:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2391726080 bytes 2023-12-02T22:30:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2418543616 bytes 2023-12-02T22:30:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2467928064 bytes 2023-12-02T22:30:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2526942208 bytes 2023-12-02T22:30:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2597463040 bytes 2023-12-02T22:30:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2618540032 bytes 2023-12-02T22:30:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2673479680 bytes 2023-12-02T22:30:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2710706176 bytes 2023-12-02T22:30:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2793444352 bytes 2023-12-02T22:30:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2812178432 bytes 2023-12-02T22:30:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2901987328 bytes 2023-12-02T22:30:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2998337536 bytes 2023-12-02T22:30:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3012466688 bytes 2023-12-02T22:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3044958208 bytes 2023-12-02T22:30:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3117396992 bytes 2023-12-02T22:30:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3161980928 bytes 2023-12-02T22:30:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3192675328 bytes 2023-12-02T22:30:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3299855360 bytes 2023-12-02T22:30:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3300489216 bytes 2023-12-02T22:30:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3329786880 bytes 2023-12-02T22:30:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3406270464 bytes 2023-12-02T22:30:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3447414784 bytes 2023-12-02T22:30:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3496540160 bytes 2023-12-02T22:30:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3568433152 bytes 2023-12-02T22:30:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3646554112 bytes 2023-12-02T22:30:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3702295552 bytes 2023-12-02T22:30:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3712732160 bytes 2023-12-02T22:30:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3755593728 bytes 2023-12-02T22:30:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3786921984 bytes 2023-12-02T22:30:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3795485696 bytes 2023-12-02T22:30:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3839240192 bytes 2023-12-02T22:30:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3843258368 bytes 2023-12-02T22:30:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3886619648 bytes 2023-12-02T22:30:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3981374464 bytes 2023-12-02T22:30:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4052815872 bytes 2023-12-02T22:30:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4053396480 bytes 2023-12-02T22:30:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4150463488 bytes 2023-12-02T22:30:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4190516224 bytes 2023-12-02T22:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4238689280 bytes 2023-12-02T22:30:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4294251520 bytes 2023-12-02T22:30:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:30:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:30:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:30:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:30:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:30:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:30:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:30:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:30:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:30:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:30:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:30:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:30:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:30:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:30:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:30:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":17850163.2,"bytes_per_snapshot":535166464.0,"last_snapshot_bytes":535504896,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556197,"remote_snapshot_timestamp":1701556197,"replay_state":"idle"} 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:30:10 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:30:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:30:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:30:11 2023-12-02T22:30:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:30:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:30:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:30:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":17850163.2,"bytes_per_snapshot":535166464.0,"last_snapshot_bytes":535504896,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556197,"remote_snapshot_timestamp":1701556197,"replay_state":"idle"} 2023-12-02T22:30:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:30:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:30:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:30:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:30:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:30:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:30:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:30:10' 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:30:10' 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:30:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:30:10' 2023-12-02T22:30:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:30:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:30:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:30:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:30:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:30:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:30:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:30:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:30:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:30:10' 2023-12-02T22:30:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:30:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:30:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:30:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:30:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap9 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2023-12-02T22:30:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:30:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:30:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:30:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:30:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:30:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:30:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:30:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:30:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:30:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:30:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:30:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:30:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:30:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:30:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:30:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:30:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:30:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:30:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:30:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:30:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:30:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:30:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:30:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:30:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:30:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:30:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 37 2023-12-02T22:30:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:30:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:30:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:30:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:30:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=37 2023-12-02T22:30:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:30:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:30:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:30:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:30:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 37 = '' 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:30:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:30:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:30:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:30:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:30:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:30:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:30:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 37 = '' 2023-12-02T22:30:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:30:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:30:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:30:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:30:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:30:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:30:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:30:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:30:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 37 = '' 2023-12-02T22:30:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:30:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:30:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:30:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:30:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=37 2023-12-02T22:30:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 37 = 37 2023-12-02T22:30:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:30:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:30:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-02T22:30:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:30:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 18 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:30:03 2023 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:30:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:30:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:30:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap8-clean ']' 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8-clean 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9 2023-12-02T22:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2023-12-02T22:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:30:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:30:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:30:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:30:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap9-clean 2023-12-02T22:30:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-02T22:30:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9-clean 2023-12-02T22:30:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:30:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:30:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T22:30:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:30:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:30:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:30:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:30:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:30:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:30:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T22:31:00.168 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9427683478801947) > weight (0.25) 2023-12-02T22:31:00.168 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6889132406741423) > weight (0.25) 2023-12-02T22:31:00.168 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T22:31:00.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T22:31:00.169 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-02T22:31:00.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:31:00.168+0000 7f1437930640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 90485) UID: 1000 2023-12-02T22:31:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 39026688 bytes 2023-12-02T22:31:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 70170624 bytes 2023-12-02T22:31:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 83229696 bytes 2023-12-02T22:31:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 119785472 bytes 2023-12-02T22:31:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 128304128 bytes 2023-12-02T22:31:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 151417856 bytes 2023-12-02T22:31:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 211697664 bytes 2023-12-02T22:31:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 223393792 bytes 2023-12-02T22:31:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 267814912 bytes 2023-12-02T22:31:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 318047232 bytes 2023-12-02T22:31:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 326315008 bytes 2023-12-02T22:31:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 430426112 bytes 2023-12-02T22:31:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 486444032 bytes 2023-12-02T22:31:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 547720192 bytes 2023-12-02T22:31:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 548359168 bytes 2023-12-02T22:31:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 579886080 bytes 2023-12-02T22:31:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 636408832 bytes 2023-12-02T22:31:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 651876352 bytes 2023-12-02T22:31:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 676168704 bytes 2023-12-02T22:31:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 709522432 bytes 2023-12-02T22:31:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 749016064 bytes 2023-12-02T22:31:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 785820672 bytes 2023-12-02T22:31:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 823767040 bytes 2023-12-02T22:31:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 866763776 bytes 2023-12-02T22:31:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 936222720 bytes 2023-12-02T22:31:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 950356992 bytes 2023-12-02T22:31:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 958287872 bytes 2023-12-02T22:31:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 973357056 bytes 2023-12-02T22:31:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1034406912 bytes 2023-12-02T22:31:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1132603392 bytes 2023-12-02T22:31:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1164174336 bytes 2023-12-02T22:31:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1186491392 bytes 2023-12-02T22:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1230575616 bytes 2023-12-02T22:31:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1249555456 bytes 2023-12-02T22:31:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1275112448 bytes 2023-12-02T22:31:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1310447616 bytes 2023-12-02T22:31:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1330723840 bytes 2023-12-02T22:31:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1352486912 bytes 2023-12-02T22:31:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1377679360 bytes 2023-12-02T22:31:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1406419968 bytes 2023-12-02T22:31:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1465690112 bytes 2023-12-02T22:31:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1519048704 bytes 2023-12-02T22:31:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1523167232 bytes 2023-12-02T22:31:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1590077440 bytes 2023-12-02T22:31:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1598879744 bytes 2023-12-02T22:31:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1620419584 bytes 2023-12-02T22:31:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1690051584 bytes 2023-12-02T22:31:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1741099008 bytes 2023-12-02T22:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1751109632 bytes 2023-12-02T22:31:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1801408512 bytes 2023-12-02T22:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1861756928 bytes 2023-12-02T22:31:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1900098560 bytes 2023-12-02T22:31:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1955008512 bytes 2023-12-02T22:31:06.170 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T22:31:06.170 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:31:06.170 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-02T22:31:06.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T22:31:06.171 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T22:31:06.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T22:31:06.173 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-02T22:31:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2005218304 bytes 2023-12-02T22:31:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2017528832 bytes 2023-12-02T22:31:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2101125120 bytes 2023-12-02T22:31:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2128305152 bytes 2023-12-02T22:31:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2145531904 bytes 2023-12-02T22:31:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2191650816 bytes 2023-12-02T22:31:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2237402112 bytes 2023-12-02T22:31:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2267035648 bytes 2023-12-02T22:31:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2283488256 bytes 2023-12-02T22:31:07.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2311874560 bytes 2023-12-02T22:31:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2344979456 bytes 2023-12-02T22:31:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2347199488 bytes 2023-12-02T22:31:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2368007168 bytes 2023-12-02T22:31:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2391503872 bytes 2023-12-02T22:31:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2438759424 bytes 2023-12-02T22:31:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2540259328 bytes 2023-12-02T22:31:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2581261312 bytes 2023-12-02T22:31:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2648005632 bytes 2023-12-02T22:31:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2726500352 bytes 2023-12-02T22:31:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2903745536 bytes 2023-12-02T22:31:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2943578112 bytes 2023-12-02T22:31:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3004040192 bytes 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:31:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:31:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:31:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:31:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T22:31:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:31:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2023-12-02T22:31:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2023-12-02T22:31:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T22:31:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T22:31:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:31:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:31:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:31:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T22:31:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:31:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:test: 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535166464.0,"last_snapshot_bytes":535504896,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556197,"remote_snapshot_timestamp":1701556197,"replay_state":"idle"} 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:30:40 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2023-12-02T22:31:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2023-12-02 22:30:41 2023-12-02T22:31:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:31:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2023-12-02T22:31:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:31:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535166464.0,"last_snapshot_bytes":535504896,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701556197,"remote_snapshot_timestamp":1701556197,"replay_state":"idle"} 2023-12-02T22:31:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:31:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:31:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2023-12-02T22:31:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:31:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:31:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2023-12-02T22:31:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.mirror_status 2023-12-02T22:31:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'last_update=2023-12-02 22:30:40' 2023-12-02T22:31:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 22:30:40' 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2023-12-02 22:30:40' 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T22:31:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:31:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror.mirror_status 2023-12-02T22:31:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T22:31:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:31:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2023-12-02T22:31:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T22:31:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2023-12-02 22:30:40' 2023-12-02T22:31:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T22:31:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2023-12-02T22:31:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2023-12-02T22:31:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2023-12-02T22:31:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap10 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2023-12-02T22:31:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:31:37.173 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T22:31:37.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T22:31:37.174 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:31:37.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:31:37.173+0000 7fc35bc51640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 93269) UID: 1000 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:31:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:31:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:31:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:31:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:31:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:31:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:31:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:31:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:31:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:31:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:31:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:31:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:31:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:31:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:31:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:31:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:31:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:31:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:31:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:31:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:31:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:31:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:31:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:31:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:31:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:31:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:31:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:31:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:31:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:31:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:31:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 41 2023-12-02T22:31:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:31:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:31:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:31:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:31:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=41 2023-12-02T22:31:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:31:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:31:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:31:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:31:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:31:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 41 = '' 2023-12-02T22:31:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:31:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:31:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:31:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:31:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 41 = '' 2023-12-02T22:31:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:31:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:31:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:31:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:31:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 41 = '' 2023-12-02T22:31:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:31:46.175 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T22:31:46.175 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:31:46.175 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T22:31:46.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:31:46.176 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:31:46.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:31:46.179 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2023-12-02T22:31:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=41 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 41 = 41 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:31:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:31:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-02T22:31:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:31:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:31:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:31:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 20 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:30:03 2023 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:31:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:31:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:31:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap9-clean ']' 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9-clean 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap10 2023-12-02T22:31:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2023-12-02T22:31:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:31:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap10-clean 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10-clean 2023-12-02T22:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T22:31:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:31:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:31:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:31:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:31:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:31:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:31:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:31:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:31:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:31:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:31:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:31:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:31:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:31:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T22:31:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:31:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:31:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:31:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:31:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:31:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:31:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:31:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:31:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-test.status 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:31:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-test.status 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T22:31:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 44 2023-12-02T22:31:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:31:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:31:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.status 2023-12-02T22:31:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T22:31:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:31:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=44 2023-12-02T22:31:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:31:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:31:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:31:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:31:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 44 = '' 2023-12-02T22:31:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:31:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:31:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:31:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:31:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:31:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:31:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 44 = '' 2023-12-02T22:31:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:31:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:31:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.status 2023-12-02T22:31:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T22:31:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:31:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:31:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=44 2023-12-02T22:31:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 44 = 44 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10-clean 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:31:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:31:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 21 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10a2b973bb60 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10a2b973bb60 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:31:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:31:48 2023 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:23:12 2023 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:31:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:31:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:31:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1-clean 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1-clean 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:31:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:31:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:31:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:31:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2-clean 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2-clean 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3-clean 2023-12-02T22:32:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-02T22:32:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3-clean 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4-clean 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4-clean 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5-clean 2023-12-02T22:32:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-02T22:32:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5-clean 2023-12-02T22:32:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2023-12-02T22:32:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6-clean 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6-clean 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7 2023-12-02T22:32:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7-clean 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7-clean 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8-clean 2023-12-02T22:32:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-02T22:32:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8-clean 2023-12-02T22:32:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9-clean 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9-clean 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10 2023-12-02T22:32:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap10 2023-12-02T22:32:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10-clean 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10-clean 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export 2023-12-02T22:32:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T22:32:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-test.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-test.export 2023-12-02T22:32:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2023-12-02T22:32:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap1 2023-12-02T22:32:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-02T22:32:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1-clean 2023-12-02T22:32:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-02T22:32:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap1-clean 2023-12-02T22:32:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-02T22:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap2 2023-12-02T22:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-02T22:32:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2-clean 2023-12-02T22:32:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-02T22:32:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap2-clean 2023-12-02T22:32:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-02T22:32:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap3 2023-12-02T22:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-02T22:32:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3-clean 2023-12-02T22:32:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-02T22:32:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap3-clean 2023-12-02T22:32:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-02T22:32:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap4 2023-12-02T22:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-02T22:32:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4-clean 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap4-clean 2023-12-02T22:32:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-02T22:32:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5 2023-12-02T22:32:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-02T22:32:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap5 2023-12-02T22:32:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-02T22:32:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5-clean 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap5-clean 2023-12-02T22:32:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-02T22:32:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap6 2023-12-02T22:32:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-02T22:32:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6-clean 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap6-clean 2023-12-02T22:32:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-02T22:32:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap7 2023-12-02T22:32:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-02T22:32:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7-clean 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap7-clean 2023-12-02T22:32:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-02T22:32:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8 2023-12-02T22:32:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-02T22:32:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap8 2023-12-02T22:32:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-02T22:32:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:46.031 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:32:46.029+0000 7f2e21bfd640 -1 reset not still connected to 0x557cdce33ee0 2023-12-02T22:32:46.031 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:32:46.029+0000 7f2e21bfd640 -1 reset not still connected to 0x557cdceba9c0 2023-12-02T22:32:46.031 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:32:46.029+0000 7f2e21bfd640 -1 reset not still connected to 0x557cdcebb110 2023-12-02T22:32:46.031 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:32:46.029+0000 7f2e21bfd640 -1 reset not still connected to 0x557cde2b0750 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8-clean 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap8-clean 2023-12-02T22:32:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-02T22:32:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap9 2023-12-02T22:32:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-02T22:32:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9-clean 2023-12-02T22:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-02T22:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap9-clean 2023-12-02T22:32:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-02T22:32:48.174 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T22:32:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T22:32:48.175 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-02T22:32:48.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:32:48.174+0000 7f69d919e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 90450) UID: 1000 2023-12-02T22:32:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap10 2023-12-02T22:32:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-02T22:32:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10-clean 2023-12-02T22:32:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-02T22:32:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap10-clean 2023-12-02T22:32:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-02T22:32:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:32:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror test 2023-12-02T22:32:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:32:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:32:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror test 2023-12-02T22:32:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:32:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-02T22:32:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-02T22:32:51.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:32:51.153+0000 7fa41485d640 -1 rbd::mirror::ImageMap: 0x55762ccf2c60 handle_peer_ack_remove: failed to remove global_image_id=e14b4c29-56f9-47b5-9ce5-53ae6fda3dc7 2023-12-02T22:32:53.180 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T22:32:53.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T22:32:53.180 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:32:53.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:32:53.179+0000 7f37bd3e8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 96835) UID: 1000 2023-12-02T22:32:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:32:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:32:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10a2b973bb60 2023-12-02T22:32:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:32:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:32:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:32:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:32:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:32:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10a2b973bb60 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10a2b973bb60 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10a2b973bb60 -o 10a2b973bb60 = 10a2b973bb60 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:32:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:32:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:32:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:32:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:32:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10a2b973bb60 2023-12-02T22:32:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10a2b973bb60 2023-12-02T22:32:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10a2b973bb60 -o 10a2b973bb60 = 10a2b973bb60 2023-12-02T22:32:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:32:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:32:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:32:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:32:56.176 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T22:32:56.176 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:32:56.176 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T22:32:56.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T22:32:56.176 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T22:32:56.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T22:32:56.180 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-02T22:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:32:56.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10a2b973bb60 2023-12-02T22:32:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10a2b973bb60 2023-12-02T22:32:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10a2b973bb60 -o 10a2b973bb60 = 10a2b973bb60 2023-12-02T22:32:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:32:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:32:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:32:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 4 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:33:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:33:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:33:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:33:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:33:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10a2b973bb60 2023-12-02T22:33:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10a2b973bb60 2023-12-02T22:33:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10a2b973bb60 -o 10a2b973bb60 = 10a2b973bb60 2023-12-02T22:33:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:33:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:33:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:33:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:33:02.181 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T22:33:02.181 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:33:02.181 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T22:33:02.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:33:02.182 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:33:02.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:33:02.185 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-02T22:33:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:33:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:33:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:33:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10a2b973bb60 2023-12-02T22:33:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10a2b973bb60 2023-12-02T22:33:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10a2b973bb60 -o 10a2b973bb60 = 10a2b973bb60 2023-12-02T22:33:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:33:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:33:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:33:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:33:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:33:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10a2b973bb60 2023-12-02T22:33:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10a2b973bb60 2023-12-02T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10a2b973bb60 -o 10a2b973bb60 = 10a2b973bb60 2023-12-02T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:33:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted 10a2b973bb60 2023-12-02T22:33:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10a2b973bb60 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T22:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:33:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: create many images' 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:33:24 TEST: create many images 2023-12-02T22:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:33:24 TEST: create many images 2023-12-02T22:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap 2023-12-02T22:33:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2023-12-02T22:33:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:33:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-02T22:33:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:33:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-02T22:33:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_1': 2023-12-02T22:33:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:33:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127d716cbdf5 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127d716cbdf5 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:24 2023 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:24 2023 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:24 2023 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 1b41b123-8764-4c2a-808d-b6f2a05a0f17 2023-12-02T22:33:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:33:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:33:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-02T22:33:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:33:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:33:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2023-12-02T22:33:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:33:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:33:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 58492928 bytes 2023-12-02T22:33:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 59096064 bytes 2023-12-02T22:33:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 105278464 bytes 2023-12-02T22:33:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 125036544 bytes 2023-12-02T22:33:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 220511232 bytes 2023-12-02T22:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 283142144 bytes 2023-12-02T22:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 307048448 bytes 2023-12-02T22:33:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 366523392 bytes 2023-12-02T22:33:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 386294784 bytes 2023-12-02T22:33:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 397737984 bytes 2023-12-02T22:33:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 398360576 bytes 2023-12-02T22:33:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 448985088 bytes 2023-12-02T22:33:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 473980928 bytes 2023-12-02T22:33:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 515717120 bytes 2023-12-02T22:33:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 543674368 bytes 2023-12-02T22:33:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 552445952 bytes 2023-12-02T22:33:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 553079808 bytes 2023-12-02T22:33:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 561412096 bytes 2023-12-02T22:33:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 610532352 bytes 2023-12-02T22:33:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 697016320 bytes 2023-12-02T22:33:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 724451328 bytes 2023-12-02T22:33:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 734998528 bytes 2023-12-02T22:33:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 735622144 bytes 2023-12-02T22:33:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 803438592 bytes 2023-12-02T22:33:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 866629632 bytes 2023-12-02T22:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 943735808 bytes 2023-12-02T22:33:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 974444544 bytes 2023-12-02T22:33:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 990970880 bytes 2023-12-02T22:33:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1027342336 bytes 2023-12-02T22:33:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1096810496 bytes 2023-12-02T22:33:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1106921472 bytes 2023-12-02T22:33:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1184115712 bytes 2023-12-02T22:33:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1233044480 bytes 2023-12-02T22:33:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1294928896 bytes 2023-12-02T22:33:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1381579776 bytes 2023-12-02T22:33:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1424262144 bytes 2023-12-02T22:33:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1435015168 bytes 2023-12-02T22:33:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1461429248 bytes 2023-12-02T22:33:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1494756352 bytes 2023-12-02T22:33:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1561649152 bytes 2023-12-02T22:33:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1598351360 bytes 2023-12-02T22:33:29.684 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:33:29.682+0000 7f4a27c08640 -1 reset not still connected to 0x55cd9603f790 2023-12-02T22:33:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1687843840 bytes 2023-12-02T22:33:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1788191744 bytes 2023-12-02T22:33:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1853727744 bytes 2023-12-02T22:33:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1909453824 bytes 2023-12-02T22:33:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1927349248 bytes 2023-12-02T22:33:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1993399296 bytes 2023-12-02T22:33:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2028116992 bytes 2023-12-02T22:33:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2071664640 bytes 2023-12-02T22:33:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2116386816 bytes 2023-12-02T22:33:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2142336000 bytes 2023-12-02T22:33:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2186956800 bytes 2023-12-02T22:33:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2201358336 bytes 2023-12-02T22:33:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2222005248 bytes 2023-12-02T22:33:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2270884864 bytes 2023-12-02T22:33:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2298883072 bytes 2023-12-02T22:33:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2316281856 bytes 2023-12-02T22:33:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2327403520 bytes 2023-12-02T22:33:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2379302912 bytes 2023-12-02T22:33:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2383858688 bytes 2023-12-02T22:33:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2471289856 bytes 2023-12-02T22:33:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2558114816 bytes 2023-12-02T22:33:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2570603520 bytes 2023-12-02T22:33:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2642545664 bytes 2023-12-02T22:33:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2680822784 bytes 2023-12-02T22:33:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2715578368 bytes 2023-12-02T22:33:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2739324928 bytes 2023-12-02T22:33:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2767769600 bytes 2023-12-02T22:33:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2807810048 bytes 2023-12-02T22:33:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2825845760 bytes 2023-12-02T22:33:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2872681472 bytes 2023-12-02T22:33:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2921682944 bytes 2023-12-02T22:33:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2953954304 bytes 2023-12-02T22:33:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2989609984 bytes 2023-12-02T22:33:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3065973760 bytes 2023-12-02T22:33:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3116335104 bytes 2023-12-02T22:33:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3137142784 bytes 2023-12-02T22:33:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3180094464 bytes 2023-12-02T22:33:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3232548864 bytes 2023-12-02T22:33:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3299883008 bytes 2023-12-02T22:33:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3368393728 bytes 2023-12-02T22:33:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3404056576 bytes 2023-12-02T22:33:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3470257152 bytes 2023-12-02T22:33:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3504149504 bytes 2023-12-02T22:33:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3551525888 bytes 2023-12-02T22:33:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3567633408 bytes 2023-12-02T22:33:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3627287552 bytes 2023-12-02T22:33:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3627872256 bytes 2023-12-02T22:33:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3635710976 bytes 2023-12-02T22:33:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3673065472 bytes 2023-12-02T22:33:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3724667904 bytes 2023-12-02T22:33:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3745756160 bytes 2023-12-02T22:33:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3807045632 bytes 2023-12-02T22:33:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3888854016 bytes 2023-12-02T22:33:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3901019136 bytes 2023-12-02T22:33:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 3922111488 bytes 2023-12-02T22:33:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 3970361344 bytes 2023-12-02T22:33:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4006936576 bytes 2023-12-02T22:33:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4050867200 bytes 2023-12-02T22:33:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4079281152 bytes 2023-12-02T22:33:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4113181696 bytes 2023-12-02T22:33:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4173605888 bytes 2023-12-02T22:33:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4208711680 bytes 2023-12-02T22:33:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4244600832 bytes 2023-12-02T22:33:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4249313280 bytes 2023-12-02T22:33:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4286526464 bytes 2023-12-02T22:33:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4347921408 bytes 2023-12-02T22:33:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10800 IOs, 4368488448 bytes 2023-12-02T22:33:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10900 IOs, 4388421632 bytes 2023-12-02T22:33:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11000 IOs, 4438353920 bytes 2023-12-02T22:33:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11100 IOs, 4496471040 bytes 2023-12-02T22:33:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11200 IOs, 4520722432 bytes 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:33:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:33:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-02T22:33:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:33:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-02T22:33:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:33:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-02T22:33:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_2': 2023-12-02T22:33:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1289476de1cf 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1289476de1cf 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:38 2023 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:38 2023 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:38 2023 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f793e7a7-0001-44a2-82ae-444e4e65879f 2023-12-02T22:33:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:33:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:33:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-02T22:33:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:33:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:33:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2023-12-02T22:33:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:33:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 36851712 bytes 2023-12-02T22:33:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 41283584 bytes 2023-12-02T22:33:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 63394816 bytes 2023-12-02T22:33:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 102885376 bytes 2023-12-02T22:33:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 119468032 bytes 2023-12-02T22:33:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 120092672 bytes 2023-12-02T22:33:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 150170624 bytes 2023-12-02T22:33:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 186365952 bytes 2023-12-02T22:33:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 305391616 bytes 2023-12-02T22:33:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 305980416 bytes 2023-12-02T22:33:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 338682880 bytes 2023-12-02T22:33:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 369943552 bytes 2023-12-02T22:33:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 395635712 bytes 2023-12-02T22:33:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 437086208 bytes 2023-12-02T22:33:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 512094208 bytes 2023-12-02T22:33:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 512699392 bytes 2023-12-02T22:33:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 526703616 bytes 2023-12-02T22:33:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 544447488 bytes 2023-12-02T22:33:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 617563136 bytes 2023-12-02T22:33:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 643488768 bytes 2023-12-02T22:33:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 670608384 bytes 2023-12-02T22:33:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 671217664 bytes 2023-12-02T22:33:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 686074880 bytes 2023-12-02T22:33:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 697268224 bytes 2023-12-02T22:33:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 722095104 bytes 2023-12-02T22:33:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 815152128 bytes 2023-12-02T22:33:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 915023872 bytes 2023-12-02T22:33:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 932575232 bytes 2023-12-02T22:33:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 962757632 bytes 2023-12-02T22:33:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 999670784 bytes 2023-12-02T22:33:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1051569152 bytes 2023-12-02T22:33:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1095506944 bytes 2023-12-02T22:33:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1105166336 bytes 2023-12-02T22:33:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1126824960 bytes 2023-12-02T22:33:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1216381952 bytes 2023-12-02T22:33:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1279076352 bytes 2023-12-02T22:33:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1351645184 bytes 2023-12-02T22:33:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1394741248 bytes 2023-12-02T22:33:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1433953280 bytes 2023-12-02T22:33:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1509099520 bytes 2023-12-02T22:33:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1587004416 bytes 2023-12-02T22:33:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1628538880 bytes 2023-12-02T22:33:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1661891584 bytes 2023-12-02T22:33:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1714315264 bytes 2023-12-02T22:33:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1800626176 bytes 2023-12-02T22:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1832885248 bytes 2023-12-02T22:33:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1870221312 bytes 2023-12-02T22:33:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1889678336 bytes 2023-12-02T22:33:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1913721856 bytes 2023-12-02T22:33:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2018396160 bytes 2023-12-02T22:33:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2081174528 bytes 2023-12-02T22:33:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2130561024 bytes 2023-12-02T22:33:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2239136768 bytes 2023-12-02T22:33:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2282563584 bytes 2023-12-02T22:33:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2308201472 bytes 2023-12-02T22:33:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2353444864 bytes 2023-12-02T22:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2417392640 bytes 2023-12-02T22:33:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2525530112 bytes 2023-12-02T22:33:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2615057408 bytes 2023-12-02T22:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2641129472 bytes 2023-12-02T22:33:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2674623488 bytes 2023-12-02T22:33:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2721416192 bytes 2023-12-02T22:33:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2746916864 bytes 2023-12-02T22:33:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2861664256 bytes 2023-12-02T22:33:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2881825792 bytes 2023-12-02T22:33:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2981937152 bytes 2023-12-02T22:33:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3044611072 bytes 2023-12-02T22:33:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3085105152 bytes 2023-12-02T22:33:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3098565632 bytes 2023-12-02T22:33:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3120022528 bytes 2023-12-02T22:33:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3140078592 bytes 2023-12-02T22:33:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3231124480 bytes 2023-12-02T22:33:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3252719616 bytes 2023-12-02T22:33:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3303371776 bytes 2023-12-02T22:33:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3334845440 bytes 2023-12-02T22:33:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3386458112 bytes 2023-12-02T22:33:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3490192384 bytes 2023-12-02T22:33:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3576042496 bytes 2023-12-02T22:33:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3591833600 bytes 2023-12-02T22:33:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3668526080 bytes 2023-12-02T22:33:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3740578816 bytes 2023-12-02T22:33:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3832519680 bytes 2023-12-02T22:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3857461248 bytes 2023-12-02T22:33:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3877134336 bytes 2023-12-02T22:33:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3906051072 bytes 2023-12-02T22:33:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3974133760 bytes 2023-12-02T22:33:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 4020270080 bytes 2023-12-02T22:33:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 4057662464 bytes 2023-12-02T22:33:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 4102168576 bytes 2023-12-02T22:33:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 4116996096 bytes 2023-12-02T22:33:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 4135497728 bytes 2023-12-02T22:33:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 4179480576 bytes 2023-12-02T22:33:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 4247647232 bytes 2023-12-02T22:33:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 4283732992 bytes 2023-12-02T22:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 4356790272 bytes 2023-12-02T22:33:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4385139712 bytes 2023-12-02T22:33:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4477232128 bytes 2023-12-02T22:33:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4548712448 bytes 2023-12-02T22:33:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4595681280 bytes 2023-12-02T22:33:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4627834880 bytes 2023-12-02T22:33:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4688322560 bytes 2023-12-02T22:33:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4702000128 bytes 2023-12-02T22:33:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4754092032 bytes 2023-12-02T22:33:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4808439808 bytes 2023-12-02T22:33:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4857664512 bytes 2023-12-02T22:33:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4908564480 bytes 2023-12-02T22:33:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4963407872 bytes 2023-12-02T22:33:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10800 IOs, 5036308480 bytes 2023-12-02T22:33:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10900 IOs, 5065046016 bytes 2023-12-02T22:33:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11000 IOs, 5073540096 bytes 2023-12-02T22:33:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11100 IOs, 5103640576 bytes 2023-12-02T22:33:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11200 IOs, 5126130688 bytes 2023-12-02T22:33:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11300 IOs, 5151064064 bytes 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:33:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:33:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:33:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-02T22:33:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:33:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-02T22:33:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:33:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_3': 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1295b151a078 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1295b151a078 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:33:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:53 2023 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:53 2023 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:53 2023 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c028f4b4-ce10-4094-9ec7-655b7f816feb 2023-12-02T22:33:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:33:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:33:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-02T22:33:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:33:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:33:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:33:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:33:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:33:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 71949312 bytes 2023-12-02T22:33:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 153491456 bytes 2023-12-02T22:33:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 205785088 bytes 2023-12-02T22:33:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 237188096 bytes 2023-12-02T22:33:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 249489408 bytes 2023-12-02T22:33:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 281909248 bytes 2023-12-02T22:33:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 303310848 bytes 2023-12-02T22:33:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 338329600 bytes 2023-12-02T22:33:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 411207680 bytes 2023-12-02T22:33:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 411825152 bytes 2023-12-02T22:33:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 435828736 bytes 2023-12-02T22:33:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 450515968 bytes 2023-12-02T22:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 454639616 bytes 2023-12-02T22:33:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 537459712 bytes 2023-12-02T22:33:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 576479232 bytes 2023-12-02T22:33:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 632165376 bytes 2023-12-02T22:33:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 662263808 bytes 2023-12-02T22:33:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 717007872 bytes 2023-12-02T22:33:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 768030720 bytes 2023-12-02T22:33:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 808105984 bytes 2023-12-02T22:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 873781248 bytes 2023-12-02T22:33:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 939298816 bytes 2023-12-02T22:33:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 944392192 bytes 2023-12-02T22:33:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1006748672 bytes 2023-12-02T22:33:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1028985856 bytes 2023-12-02T22:33:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1060183040 bytes 2023-12-02T22:33:57.181 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.556899682408698) > weight (0.25) 2023-12-02T22:33:57.181 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.795420500536301) > weight (0.25) 2023-12-02T22:33:57.181 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T22:33:57.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T22:33:57.181 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T22:33:57.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:33:57.181+0000 7fa41585f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 96514) UID: 1000 2023-12-02T22:33:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1108714496 bytes 2023-12-02T22:33:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1183686656 bytes 2023-12-02T22:33:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1199796224 bytes 2023-12-02T22:33:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1252648960 bytes 2023-12-02T22:33:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1265195008 bytes 2023-12-02T22:33:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1347861504 bytes 2023-12-02T22:33:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1423955968 bytes 2023-12-02T22:33:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1488863232 bytes 2023-12-02T22:33:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1489471488 bytes 2023-12-02T22:33:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1530426368 bytes 2023-12-02T22:33:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1568105472 bytes 2023-12-02T22:33:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1627850752 bytes 2023-12-02T22:33:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1634237440 bytes 2023-12-02T22:33:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1719693312 bytes 2023-12-02T22:33:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1764471808 bytes 2023-12-02T22:33:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:33:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:33:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:33:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:33:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:33:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-02T22:34:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-02T22:34:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_4': 2023-12-02T22:34:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a131e72aa 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a131e72aa 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:59 2023 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:59 2023 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:59 2023 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a8f37621-7a4e-4f6c-b01d-f20a1d76f088 2023-12-02T22:34:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-02T22:34:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:34:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2023-12-02T22:34:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 61637632 bytes 2023-12-02T22:34:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 62242816 bytes 2023-12-02T22:34:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 63696896 bytes 2023-12-02T22:34:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 81524736 bytes 2023-12-02T22:34:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 108592128 bytes 2023-12-02T22:34:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 162490368 bytes 2023-12-02T22:34:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 175156224 bytes 2023-12-02T22:34:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 218907648 bytes 2023-12-02T22:34:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 300846080 bytes 2023-12-02T22:34:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 377346048 bytes 2023-12-02T22:34:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 390011904 bytes 2023-12-02T22:34:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 448789504 bytes 2023-12-02T22:34:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 488529920 bytes 2023-12-02T22:34:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 517786624 bytes 2023-12-02T22:34:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 621731840 bytes 2023-12-02T22:34:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 622333952 bytes 2023-12-02T22:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 643063808 bytes 2023-12-02T22:34:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 699628544 bytes 2023-12-02T22:34:02.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 718348288 bytes 2023-12-02T22:34:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 790181888 bytes 2023-12-02T22:34:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 823892992 bytes 2023-12-02T22:34:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 835366912 bytes 2023-12-02T22:34:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 835940352 bytes 2023-12-02T22:34:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 909329408 bytes 2023-12-02T22:34:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 927885312 bytes 2023-12-02T22:34:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 956965888 bytes 2023-12-02T22:34:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 986398720 bytes 2023-12-02T22:34:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1064148992 bytes 2023-12-02T22:34:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1091913728 bytes 2023-12-02T22:34:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1155437568 bytes 2023-12-02T22:34:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1192125440 bytes 2023-12-02T22:34:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1231145984 bytes 2023-12-02T22:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1263187968 bytes 2023-12-02T22:34:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1326760960 bytes 2023-12-02T22:34:04.182 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T22:34:04.182 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:34:04.182 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-02T22:34:04.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T22:34:04.182 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T22:34:04.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T22:34:04.185 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-02T22:34:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1421110272 bytes 2023-12-02T22:34:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1459778560 bytes 2023-12-02T22:34:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1533343744 bytes 2023-12-02T22:34:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1540102144 bytes 2023-12-02T22:34:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1621542912 bytes 2023-12-02T22:34:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1622160384 bytes 2023-12-02T22:34:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1648568320 bytes 2023-12-02T22:34:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1680155648 bytes 2023-12-02T22:34:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1718424576 bytes 2023-12-02T22:34:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1772625920 bytes 2023-12-02T22:34:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1781990400 bytes 2023-12-02T22:34:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1829098496 bytes 2023-12-02T22:34:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1829705728 bytes 2023-12-02T22:34:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1853841408 bytes 2023-12-02T22:34:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1893817344 bytes 2023-12-02T22:34:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1894424576 bytes 2023-12-02T22:34:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1925401600 bytes 2023-12-02T22:34:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1997108224 bytes 2023-12-02T22:34:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1997736960 bytes 2023-12-02T22:34:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2043208704 bytes 2023-12-02T22:34:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2061784064 bytes 2023-12-02T22:34:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2122202112 bytes 2023-12-02T22:34:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2178115584 bytes 2023-12-02T22:34:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2211391488 bytes 2023-12-02T22:34:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2246536192 bytes 2023-12-02T22:34:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2267715584 bytes 2023-12-02T22:34:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2345457664 bytes 2023-12-02T22:34:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2379530240 bytes 2023-12-02T22:34:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2404590592 bytes 2023-12-02T22:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2465929216 bytes 2023-12-02T22:34:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2495854592 bytes 2023-12-02T22:34:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2530223104 bytes 2023-12-02T22:34:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2618994688 bytes 2023-12-02T22:34:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2643512320 bytes 2023-12-02T22:34:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2692816896 bytes 2023-12-02T22:34:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2798568448 bytes 2023-12-02T22:34:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2863710208 bytes 2023-12-02T22:34:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2943428608 bytes 2023-12-02T22:34:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2986568704 bytes 2023-12-02T22:34:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3010422784 bytes 2023-12-02T22:34:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3063337984 bytes 2023-12-02T22:34:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3101808640 bytes 2023-12-02T22:34:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3154640896 bytes 2023-12-02T22:34:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3212656640 bytes 2023-12-02T22:34:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3253923840 bytes 2023-12-02T22:34:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3260474368 bytes 2023-12-02T22:34:09.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3323991040 bytes 2023-12-02T22:34:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3427047424 bytes 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2023-12-02T22:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-02T22:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-02T22:34:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-02T22:34:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_5': 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12b2df252827 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12b2df252827 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:10 2023 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:10 2023 2023-12-02T22:34:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:10 2023 2023-12-02T22:34:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 3083f507-4484-4113-938a-b4257c974175 2023-12-02T22:34:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-02T22:34:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:34:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:34:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 59452416 bytes 2023-12-02T22:34:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 90610688 bytes 2023-12-02T22:34:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 140981248 bytes 2023-12-02T22:34:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 141593600 bytes 2023-12-02T22:34:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 200423424 bytes 2023-12-02T22:34:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 242038784 bytes 2023-12-02T22:34:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 267821056 bytes 2023-12-02T22:34:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 392559616 bytes 2023-12-02T22:34:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 415819776 bytes 2023-12-02T22:34:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 446879744 bytes 2023-12-02T22:34:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 502367232 bytes 2023-12-02T22:34:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 525715456 bytes 2023-12-02T22:34:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 568572928 bytes 2023-12-02T22:34:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 630913024 bytes 2023-12-02T22:34:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 684538880 bytes 2023-12-02T22:34:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 728988672 bytes 2023-12-02T22:34:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 774236160 bytes 2023-12-02T22:34:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 828393472 bytes 2023-12-02T22:34:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 960038912 bytes 2023-12-02T22:34:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 981507072 bytes 2023-12-02T22:34:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1022558208 bytes 2023-12-02T22:34:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1029170176 bytes 2023-12-02T22:34:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1036480512 bytes 2023-12-02T22:34:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1058648064 bytes 2023-12-02T22:34:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1090141184 bytes 2023-12-02T22:34:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1123444736 bytes 2023-12-02T22:34:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1127041024 bytes 2023-12-02T22:34:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1135256576 bytes 2023-12-02T22:34:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1173980160 bytes 2023-12-02T22:34:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1213485056 bytes 2023-12-02T22:34:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1272621056 bytes 2023-12-02T22:34:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1320154112 bytes 2023-12-02T22:34:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1374881792 bytes 2023-12-02T22:34:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1375491072 bytes 2023-12-02T22:34:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1476995072 bytes 2023-12-02T22:34:15.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6299060625407957) > weight (0.25) 2023-12-02T22:34:15.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8504389004098032) > weight (0.25) 2023-12-02T22:34:15.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7085952006757111) > weight (0.25) 2023-12-02T22:34:15.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5831337892196337) > weight (0.25) 2023-12-02T22:34:15.186 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-12-02T22:34:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1534015488 bytes 2023-12-02T22:34:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1577526272 bytes 2023-12-02T22:34:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1592747008 bytes 2023-12-02T22:34:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1678621696 bytes 2023-12-02T22:34:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1716215808 bytes 2023-12-02T22:34:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1726271488 bytes 2023-12-02T22:34:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1739201536 bytes 2023-12-02T22:34:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1778068480 bytes 2023-12-02T22:34:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1843749888 bytes 2023-12-02T22:34:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1864949760 bytes 2023-12-02T22:34:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1879064576 bytes 2023-12-02T22:34:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1909122048 bytes 2023-12-02T22:34:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1957693440 bytes 2023-12-02T22:34:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1975233536 bytes 2023-12-02T22:34:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1995561984 bytes 2023-12-02T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2067973120 bytes 2023-12-02T22:34:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2117216256 bytes 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-02T22:34:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_6': 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12bea111bb53 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12bea111bb53 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:17 2023 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:17 2023 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:17 2023 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 6f4a39a9-03a2-4b4f-a1b0-8e15b573f1ab 2023-12-02T22:34:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-02T22:34:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:34:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:34:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 81521664 bytes 2023-12-02T22:34:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 116247552 bytes 2023-12-02T22:34:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 163007488 bytes 2023-12-02T22:34:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 222373888 bytes 2023-12-02T22:34:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 249361408 bytes 2023-12-02T22:34:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 272223232 bytes 2023-12-02T22:34:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 316479488 bytes 2023-12-02T22:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 373126144 bytes 2023-12-02T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 418909184 bytes 2023-12-02T22:34:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 498043904 bytes 2023-12-02T22:34:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 555209728 bytes 2023-12-02T22:34:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 635361280 bytes 2023-12-02T22:34:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 660624384 bytes 2023-12-02T22:34:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 714536960 bytes 2023-12-02T22:34:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 758794240 bytes 2023-12-02T22:34:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 842756096 bytes 2023-12-02T22:34:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 898298880 bytes 2023-12-02T22:34:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 976417792 bytes 2023-12-02T22:34:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 1037847552 bytes 2023-12-02T22:34:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 1074847744 bytes 2023-12-02T22:34:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1075477504 bytes 2023-12-02T22:34:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1123615744 bytes 2023-12-02T22:34:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1184648192 bytes 2023-12-02T22:34:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1206896640 bytes 2023-12-02T22:34:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1237018624 bytes 2023-12-02T22:34:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1283014656 bytes 2023-12-02T22:34:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1330518016 bytes 2023-12-02T22:34:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1361781760 bytes 2023-12-02T22:34:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1380291584 bytes 2023-12-02T22:34:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1431268352 bytes 2023-12-02T22:34:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1514045440 bytes 2023-12-02T22:34:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1590769664 bytes 2023-12-02T22:34:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1659302912 bytes 2023-12-02T22:34:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1677842432 bytes 2023-12-02T22:34:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1709509632 bytes 2023-12-02T22:34:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1794204672 bytes 2023-12-02T22:34:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1864256512 bytes 2023-12-02T22:34:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1864871936 bytes 2023-12-02T22:34:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:34:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-02T22:34:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-02T22:34:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-02T22:34:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_7': 2023-12-02T22:34:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12ca8ba9f593 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12ca8ba9f593 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:23 2023 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:23 2023 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:23 2023 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 29d0f358-af7a-4fe0-84a4-8618fbdbda52 2023-12-02T22:34:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-02T22:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:34:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 15712256 bytes 2023-12-02T22:34:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 107429888 bytes 2023-12-02T22:34:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 130177024 bytes 2023-12-02T22:34:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 180201472 bytes 2023-12-02T22:34:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 211854336 bytes 2023-12-02T22:34:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 289922048 bytes 2023-12-02T22:34:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 311968768 bytes 2023-12-02T22:34:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 323224576 bytes 2023-12-02T22:34:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 369867776 bytes 2023-12-02T22:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 401494016 bytes 2023-12-02T22:34:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 426923008 bytes 2023-12-02T22:34:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 454990848 bytes 2023-12-02T22:34:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 513205248 bytes 2023-12-02T22:34:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 592650240 bytes 2023-12-02T22:34:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 648099840 bytes 2023-12-02T22:34:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 674036736 bytes 2023-12-02T22:34:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 740092928 bytes 2023-12-02T22:34:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 800215040 bytes 2023-12-02T22:34:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 820126720 bytes 2023-12-02T22:34:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 840072192 bytes 2023-12-02T22:34:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 867784704 bytes 2023-12-02T22:34:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 936527872 bytes 2023-12-02T22:34:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 993245184 bytes 2023-12-02T22:34:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1088111616 bytes 2023-12-02T22:34:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1145078784 bytes 2023-12-02T22:34:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1229441024 bytes 2023-12-02T22:34:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1299537920 bytes 2023-12-02T22:34:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1369074688 bytes 2023-12-02T22:34:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1425707008 bytes 2023-12-02T22:34:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1491989504 bytes 2023-12-02T22:34:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1516698624 bytes 2023-12-02T22:34:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1656120320 bytes 2023-12-02T22:34:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1685896192 bytes 2023-12-02T22:34:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1728287744 bytes 2023-12-02T22:34:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1759716352 bytes 2023-12-02T22:34:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1814451200 bytes 2023-12-02T22:34:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1877320704 bytes 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-02T22:34:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-02T22:34:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_8': 2023-12-02T22:34:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12d6bd3b0273 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12d6bd3b0273 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:28 2023 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:28 2023 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:28 2023 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 54d6e2fd-a8c4-41f2-93f4-8602f730be4d 2023-12-02T22:34:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-02T22:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2023-12-02T22:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 44392448 bytes 2023-12-02T22:34:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 67283968 bytes 2023-12-02T22:34:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 85639168 bytes 2023-12-02T22:34:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 126302208 bytes 2023-12-02T22:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 126939136 bytes 2023-12-02T22:34:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 174094336 bytes 2023-12-02T22:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 250775552 bytes 2023-12-02T22:34:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 292353024 bytes 2023-12-02T22:34:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 310719488 bytes 2023-12-02T22:34:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 411376640 bytes 2023-12-02T22:34:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 435771392 bytes 2023-12-02T22:34:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 473651200 bytes 2023-12-02T22:34:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 557667328 bytes 2023-12-02T22:34:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 582720512 bytes 2023-12-02T22:34:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 601180160 bytes 2023-12-02T22:34:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 652404736 bytes 2023-12-02T22:34:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 664466432 bytes 2023-12-02T22:34:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 732273664 bytes 2023-12-02T22:34:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 809500672 bytes 2023-12-02T22:34:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 844830720 bytes 2023-12-02T22:34:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 890578944 bytes 2023-12-02T22:34:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 939705344 bytes 2023-12-02T22:34:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 969548800 bytes 2023-12-02T22:34:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 987779072 bytes 2023-12-02T22:34:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1030841344 bytes 2023-12-02T22:34:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1159145472 bytes 2023-12-02T22:34:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1227691008 bytes 2023-12-02T22:34:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1265397760 bytes 2023-12-02T22:34:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1298336768 bytes 2023-12-02T22:34:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1354046464 bytes 2023-12-02T22:34:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1368104960 bytes 2023-12-02T22:34:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1384664064 bytes 2023-12-02T22:34:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1451029504 bytes 2023-12-02T22:34:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1500564480 bytes 2023-12-02T22:34:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1512755200 bytes 2023-12-02T22:34:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1534063616 bytes 2023-12-02T22:34:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1554024448 bytes 2023-12-02T22:34:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1562766336 bytes 2023-12-02T22:34:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1596899328 bytes 2023-12-02T22:34:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1627270144 bytes 2023-12-02T22:34:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1676510208 bytes 2023-12-02T22:34:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1727993856 bytes 2023-12-02T22:34:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1784221696 bytes 2023-12-02T22:34:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1822137344 bytes 2023-12-02T22:34:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1853377536 bytes 2023-12-02T22:34:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1870756864 bytes 2023-12-02T22:34:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1880731648 bytes 2023-12-02T22:34:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1898071040 bytes 2023-12-02T22:34:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1982264320 bytes 2023-12-02T22:34:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2055101440 bytes 2023-12-02T22:34:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2156866560 bytes 2023-12-02T22:34:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2196259840 bytes 2023-12-02T22:34:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2210085888 bytes 2023-12-02T22:34:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2291924992 bytes 2023-12-02T22:34:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2353089536 bytes 2023-12-02T22:34:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2387542016 bytes 2023-12-02T22:34:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2411751424 bytes 2023-12-02T22:34:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2450233344 bytes 2023-12-02T22:34:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2464262144 bytes 2023-12-02T22:34:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2506555392 bytes 2023-12-02T22:34:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2575879168 bytes 2023-12-02T22:34:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2629023744 bytes 2023-12-02T22:34:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2662942720 bytes 2023-12-02T22:34:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2709516288 bytes 2023-12-02T22:34:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2714700800 bytes 2023-12-02T22:34:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2775747584 bytes 2023-12-02T22:34:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2802377728 bytes 2023-12-02T22:34:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2812529664 bytes 2023-12-02T22:34:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2917795840 bytes 2023-12-02T22:34:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2943400960 bytes 2023-12-02T22:34:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3022246912 bytes 2023-12-02T22:34:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3102342144 bytes 2023-12-02T22:34:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3136076800 bytes 2023-12-02T22:34:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3183241216 bytes 2023-12-02T22:34:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3198546944 bytes 2023-12-02T22:34:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3292670976 bytes 2023-12-02T22:34:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3341737984 bytes 2023-12-02T22:34:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3368182784 bytes 2023-12-02T22:34:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3415753728 bytes 2023-12-02T22:34:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3448652800 bytes 2023-12-02T22:34:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3469883392 bytes 2023-12-02T22:34:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3539210240 bytes 2023-12-02T22:34:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3550107648 bytes 2023-12-02T22:34:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3550732288 bytes 2023-12-02T22:34:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3582711808 bytes 2023-12-02T22:34:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3621444608 bytes 2023-12-02T22:34:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3642337280 bytes 2023-12-02T22:34:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:34:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-02T22:34:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-02T22:34:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_9': 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12e2dc8e2841 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12e2dc8e2841 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:40 2023 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:40 2023 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:40 2023 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f162a3b0-3c88-4e6c-ae84-ffb10c73c491 2023-12-02T22:34:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-02T22:34:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:34:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:34:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 9255936 bytes 2023-12-02T22:34:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 87310336 bytes 2023-12-02T22:34:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 139855872 bytes 2023-12-02T22:34:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 207882240 bytes 2023-12-02T22:34:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 246335488 bytes 2023-12-02T22:34:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 250326016 bytes 2023-12-02T22:34:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 277294080 bytes 2023-12-02T22:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 362604544 bytes 2023-12-02T22:34:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 395376640 bytes 2023-12-02T22:34:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 399171584 bytes 2023-12-02T22:34:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 413069312 bytes 2023-12-02T22:34:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 432473088 bytes 2023-12-02T22:34:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 440936448 bytes 2023-12-02T22:34:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 477895680 bytes 2023-12-02T22:34:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 518732800 bytes 2023-12-02T22:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 519795712 bytes 2023-12-02T22:34:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 554835968 bytes 2023-12-02T22:34:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 593854464 bytes 2023-12-02T22:34:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 674799616 bytes 2023-12-02T22:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 741066752 bytes 2023-12-02T22:34:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 776142848 bytes 2023-12-02T22:34:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 824432640 bytes 2023-12-02T22:34:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 844593152 bytes 2023-12-02T22:34:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 871088128 bytes 2023-12-02T22:34:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 946519040 bytes 2023-12-02T22:34:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 970959872 bytes 2023-12-02T22:34:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 988160000 bytes 2023-12-02T22:34:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1058377728 bytes 2023-12-02T22:34:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1077442560 bytes 2023-12-02T22:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1078043648 bytes 2023-12-02T22:34:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1145531392 bytes 2023-12-02T22:34:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1178012672 bytes 2023-12-02T22:34:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1239084032 bytes 2023-12-02T22:34:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1313556480 bytes 2023-12-02T22:34:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1332329472 bytes 2023-12-02T22:34:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1386936320 bytes 2023-12-02T22:34:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1437298688 bytes 2023-12-02T22:34:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1493094400 bytes 2023-12-02T22:34:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1557182464 bytes 2023-12-02T22:34:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1560067072 bytes 2023-12-02T22:34:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1595835392 bytes 2023-12-02T22:34:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-02T22:34:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-02T22:34:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_10': 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12eed8d78fc3 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12eed8d78fc3 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:46 2023 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:46 2023 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:46 2023 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 744cfe14-46e0-4471-aa62-cd6ed74706ca 2023-12-02T22:34:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-02T22:34:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:34:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:34:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 56814592 bytes 2023-12-02T22:34:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 73458688 bytes 2023-12-02T22:34:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 104554496 bytes 2023-12-02T22:34:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 134564864 bytes 2023-12-02T22:34:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 153307136 bytes 2023-12-02T22:34:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 159244288 bytes 2023-12-02T22:34:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 181843968 bytes 2023-12-02T22:34:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 206825472 bytes 2023-12-02T22:34:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 260013056 bytes 2023-12-02T22:34:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 285135872 bytes 2023-12-02T22:34:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 306642944 bytes 2023-12-02T22:34:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 307274752 bytes 2023-12-02T22:34:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 427446272 bytes 2023-12-02T22:34:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 444761088 bytes 2023-12-02T22:34:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 481338368 bytes 2023-12-02T22:34:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 554792960 bytes 2023-12-02T22:34:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 573402112 bytes 2023-12-02T22:34:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 615593984 bytes 2023-12-02T22:34:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 655282176 bytes 2023-12-02T22:34:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 680456192 bytes 2023-12-02T22:34:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 713939968 bytes 2023-12-02T22:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 736869376 bytes 2023-12-02T22:34:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 790130688 bytes 2023-12-02T22:34:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 860493824 bytes 2023-12-02T22:34:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 874515456 bytes 2023-12-02T22:34:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 942757888 bytes 2023-12-02T22:34:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1003815936 bytes 2023-12-02T22:34:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1051768832 bytes 2023-12-02T22:34:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1077385216 bytes 2023-12-02T22:34:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1159326720 bytes 2023-12-02T22:34:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1200848896 bytes 2023-12-02T22:34:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1207414784 bytes 2023-12-02T22:34:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1226332160 bytes 2023-12-02T22:34:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1298007040 bytes 2023-12-02T22:34:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1433999360 bytes 2023-12-02T22:34:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1485950976 bytes 2023-12-02T22:34:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1561399296 bytes 2023-12-02T22:34:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1567297536 bytes 2023-12-02T22:34:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1601003520 bytes 2023-12-02T22:34:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1626904576 bytes 2023-12-02T22:34:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1660518400 bytes 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:34:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-12-02T22:34:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-02T22:34:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:34:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-02T22:34:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-02T22:34:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_11': 2023-12-02T22:34:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12fa1bf15a2b 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12fa1bf15a2b 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:52 2023 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:52 2023 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:52 2023 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7d477ce5-b8e7-48b3-aa17-29ac3ee8f70c 2023-12-02T22:34:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-02T22:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:34:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 30300160 bytes 2023-12-02T22:34:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 81243136 bytes 2023-12-02T22:34:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 116706304 bytes 2023-12-02T22:34:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 120005632 bytes 2023-12-02T22:34:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 175196160 bytes 2023-12-02T22:34:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 198961152 bytes 2023-12-02T22:34:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 237106176 bytes 2023-12-02T22:34:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 260902912 bytes 2023-12-02T22:34:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 300306432 bytes 2023-12-02T22:34:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 361033728 bytes 2023-12-02T22:34:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 397620224 bytes 2023-12-02T22:34:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 472530944 bytes 2023-12-02T22:34:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 526747648 bytes 2023-12-02T22:34:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 581134336 bytes 2023-12-02T22:34:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 628150272 bytes 2023-12-02T22:34:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 664055808 bytes 2023-12-02T22:34:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 664639488 bytes 2023-12-02T22:34:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 691670016 bytes 2023-12-02T22:34:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 692269056 bytes 2023-12-02T22:34:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 748802048 bytes 2023-12-02T22:34:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 811694080 bytes 2023-12-02T22:34:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 832080896 bytes 2023-12-02T22:34:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 853028864 bytes 2023-12-02T22:34:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 866577408 bytes 2023-12-02T22:34:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 919731200 bytes 2023-12-02T22:34:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 946350080 bytes 2023-12-02T22:34:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1000799232 bytes 2023-12-02T22:34:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1074191360 bytes 2023-12-02T22:34:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1098081280 bytes 2023-12-02T22:34:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1143313408 bytes 2023-12-02T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1184867328 bytes 2023-12-02T22:34:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1215094784 bytes 2023-12-02T22:34:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1254453248 bytes 2023-12-02T22:34:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1275418624 bytes 2023-12-02T22:34:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1300404224 bytes 2023-12-02T22:34:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1328809984 bytes 2023-12-02T22:34:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1397911552 bytes 2023-12-02T22:34:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1446538240 bytes 2023-12-02T22:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1455604736 bytes 2023-12-02T22:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1528939520 bytes 2023-12-02T22:34:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:34:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:34:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:34:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:34:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:34:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:34:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:34:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:34:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-02T22:34:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:34:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-02T22:34:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_12': 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1306be58fa22 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1306be58fa22 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:58 2023 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:58 2023 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:58 2023 2023-12-02T22:34:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:34:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:34:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 87b8efc0-f045-4058-89b2-d6580e15413a 2023-12-02T22:34:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:34:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:34:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-02T22:34:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:34:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:34:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:34:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:34:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:34:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:34:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:34:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 617472 bytes 2023-12-02T22:34:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 27028480 bytes 2023-12-02T22:34:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 125634560 bytes 2023-12-02T22:34:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 134026240 bytes 2023-12-02T22:34:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 161527808 bytes 2023-12-02T22:34:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 196888576 bytes 2023-12-02T22:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 256371712 bytes 2023-12-02T22:35:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 286354432 bytes 2023-12-02T22:35:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 333955072 bytes 2023-12-02T22:35:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 418759680 bytes 2023-12-02T22:35:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 466942976 bytes 2023-12-02T22:35:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 500966400 bytes 2023-12-02T22:35:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 515521536 bytes 2023-12-02T22:35:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 552946688 bytes 2023-12-02T22:35:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 566793216 bytes 2023-12-02T22:35:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 615929856 bytes 2023-12-02T22:35:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 683570176 bytes 2023-12-02T22:35:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 738179072 bytes 2023-12-02T22:35:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 784310272 bytes 2023-12-02T22:35:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 796767232 bytes 2023-12-02T22:35:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 823064576 bytes 2023-12-02T22:35:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 880778240 bytes 2023-12-02T22:35:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 918849536 bytes 2023-12-02T22:35:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1023244288 bytes 2023-12-02T22:35:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1023856640 bytes 2023-12-02T22:35:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1043436544 bytes 2023-12-02T22:35:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1060920320 bytes 2023-12-02T22:35:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1117825024 bytes 2023-12-02T22:35:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1128460288 bytes 2023-12-02T22:35:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1177796608 bytes 2023-12-02T22:35:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1205801984 bytes 2023-12-02T22:35:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1228240896 bytes 2023-12-02T22:35:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1253801984 bytes 2023-12-02T22:35:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1331137536 bytes 2023-12-02T22:35:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1373791232 bytes 2023-12-02T22:35:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1451739136 bytes 2023-12-02T22:35:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1498340352 bytes 2023-12-02T22:35:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1563145216 bytes 2023-12-02T22:35:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1608946688 bytes 2023-12-02T22:35:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1623451648 bytes 2023-12-02T22:35:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1639490560 bytes 2023-12-02T22:35:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1703375872 bytes 2023-12-02T22:35:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1703984128 bytes 2023-12-02T22:35:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1762036736 bytes 2023-12-02T22:35:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1773865984 bytes 2023-12-02T22:35:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1835703296 bytes 2023-12-02T22:35:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1895370752 bytes 2023-12-02T22:35:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1895975936 bytes 2023-12-02T22:35:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1981484032 bytes 2023-12-02T22:35:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1989482496 bytes 2023-12-02T22:35:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2007904256 bytes 2023-12-02T22:35:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2047596544 bytes 2023-12-02T22:35:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2116572160 bytes 2023-12-02T22:35:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2201922560 bytes 2023-12-02T22:35:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2231835648 bytes 2023-12-02T22:35:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2273264640 bytes 2023-12-02T22:35:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2328702976 bytes 2023-12-02T22:35:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2362366976 bytes 2023-12-02T22:35:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2377622528 bytes 2023-12-02T22:35:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2457563136 bytes 2023-12-02T22:35:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2472853504 bytes 2023-12-02T22:35:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2485421056 bytes 2023-12-02T22:35:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2548953088 bytes 2023-12-02T22:35:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2589636608 bytes 2023-12-02T22:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2691332096 bytes 2023-12-02T22:35:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2730888192 bytes 2023-12-02T22:35:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2755284992 bytes 2023-12-02T22:35:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2772374528 bytes 2023-12-02T22:35:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2843881472 bytes 2023-12-02T22:35:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2859623424 bytes 2023-12-02T22:35:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2901065728 bytes 2023-12-02T22:35:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2971498496 bytes 2023-12-02T22:35:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3001711616 bytes 2023-12-02T22:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3006120960 bytes 2023-12-02T22:35:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3053313024 bytes 2023-12-02T22:35:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3089412096 bytes 2023-12-02T22:35:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3121232896 bytes 2023-12-02T22:35:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3152546816 bytes 2023-12-02T22:35:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3184508928 bytes 2023-12-02T22:35:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3222905856 bytes 2023-12-02T22:35:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3249898496 bytes 2023-12-02T22:35:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3313004544 bytes 2023-12-02T22:35:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3363061760 bytes 2023-12-02T22:35:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3433460736 bytes 2023-12-02T22:35:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3506876416 bytes 2023-12-02T22:35:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3553102848 bytes 2023-12-02T22:35:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3587603456 bytes 2023-12-02T22:35:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3638312960 bytes 2023-12-02T22:35:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3702407168 bytes 2023-12-02T22:35:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3710682112 bytes 2023-12-02T22:35:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3724199936 bytes 2023-12-02T22:35:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3775951872 bytes 2023-12-02T22:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3797882880 bytes 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2023-12-02T22:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-12-02T22:35:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-02T22:35:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:35:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-02T22:35:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_13': 2023-12-02T22:35:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:35:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:35:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:35:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1312f9eb2adf 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1312f9eb2adf 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:11 2023 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:11 2023 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:11 2023 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d76b9c8b-eb85-4586-95f4-981ca6150274 2023-12-02T22:35:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:35:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:35:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-02T22:35:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:35:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:35:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 50295808 bytes 2023-12-02T22:35:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 61495296 bytes 2023-12-02T22:35:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 114612224 bytes 2023-12-02T22:35:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 191845376 bytes 2023-12-02T22:35:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 205419520 bytes 2023-12-02T22:35:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 253414400 bytes 2023-12-02T22:35:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 279019520 bytes 2023-12-02T22:35:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 352063488 bytes 2023-12-02T22:35:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 364917760 bytes 2023-12-02T22:35:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 403061760 bytes 2023-12-02T22:35:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 434776064 bytes 2023-12-02T22:35:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 476694528 bytes 2023-12-02T22:35:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 544818176 bytes 2023-12-02T22:35:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 637704192 bytes 2023-12-02T22:35:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 654147584 bytes 2023-12-02T22:35:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 677206016 bytes 2023-12-02T22:35:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 724280320 bytes 2023-12-02T22:35:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 731044864 bytes 2023-12-02T22:35:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 748788736 bytes 2023-12-02T22:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 778904576 bytes 2023-12-02T22:35:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 799250432 bytes 2023-12-02T22:35:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 819300352 bytes 2023-12-02T22:35:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 928724992 bytes 2023-12-02T22:35:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 970369024 bytes 2023-12-02T22:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1014996992 bytes 2023-12-02T22:35:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1020072960 bytes 2023-12-02T22:35:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1113273344 bytes 2023-12-02T22:35:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1177066496 bytes 2023-12-02T22:35:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1231152128 bytes 2023-12-02T22:35:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1271843840 bytes 2023-12-02T22:35:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1305987072 bytes 2023-12-02T22:35:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1392890880 bytes 2023-12-02T22:35:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1444134912 bytes 2023-12-02T22:35:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1478305792 bytes 2023-12-02T22:35:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1559907328 bytes 2023-12-02T22:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1600397312 bytes 2023-12-02T22:35:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1650607104 bytes 2023-12-02T22:35:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1653487616 bytes 2023-12-02T22:35:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1683350528 bytes 2023-12-02T22:35:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1699877888 bytes 2023-12-02T22:35:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1722420224 bytes 2023-12-02T22:35:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1752473600 bytes 2023-12-02T22:35:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1786446848 bytes 2023-12-02T22:35:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1813890048 bytes 2023-12-02T22:35:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1907181568 bytes 2023-12-02T22:35:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1946114048 bytes 2023-12-02T22:35:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1977825280 bytes 2023-12-02T22:35:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2022615040 bytes 2023-12-02T22:35:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2047036416 bytes 2023-12-02T22:35:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2100323328 bytes 2023-12-02T22:35:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2141820928 bytes 2023-12-02T22:35:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2172518400 bytes 2023-12-02T22:35:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2259652608 bytes 2023-12-02T22:35:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2323982336 bytes 2023-12-02T22:35:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2339706880 bytes 2023-12-02T22:35:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2397688832 bytes 2023-12-02T22:35:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2454509568 bytes 2023-12-02T22:35:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2475731968 bytes 2023-12-02T22:35:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2490567680 bytes 2023-12-02T22:35:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2514106368 bytes 2023-12-02T22:35:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2581739520 bytes 2023-12-02T22:35:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2636171264 bytes 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:35:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:35:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:35:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:35:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-02T22:35:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:35:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-02T22:35:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_14': 2023-12-02T22:35:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 131e3b658c57 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.131e3b658c57 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:20 2023 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:20 2023 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:20 2023 2023-12-02T22:35:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:35:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:35:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: da7e5eab-bc9e-4019-826f-9d3346e2d46b 2023-12-02T22:35:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:35:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:35:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-02T22:35:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:35:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:35:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2023-12-02T22:35:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:35:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:35:21.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 45656064 bytes 2023-12-02T22:35:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 65193984 bytes 2023-12-02T22:35:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 77180928 bytes 2023-12-02T22:35:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 117169152 bytes 2023-12-02T22:35:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 144015360 bytes 2023-12-02T22:35:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 206589952 bytes 2023-12-02T22:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 249438208 bytes 2023-12-02T22:35:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 272963584 bytes 2023-12-02T22:35:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 315364352 bytes 2023-12-02T22:35:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 351111168 bytes 2023-12-02T22:35:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 386290688 bytes 2023-12-02T22:35:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 412894208 bytes 2023-12-02T22:35:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 445427712 bytes 2023-12-02T22:35:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 509566976 bytes 2023-12-02T22:35:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 535063552 bytes 2023-12-02T22:35:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 548344832 bytes 2023-12-02T22:35:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 565547008 bytes 2023-12-02T22:35:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 574955520 bytes 2023-12-02T22:35:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 654582784 bytes 2023-12-02T22:35:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 696806400 bytes 2023-12-02T22:35:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 730133504 bytes 2023-12-02T22:35:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 778488832 bytes 2023-12-02T22:35:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 810180608 bytes 2023-12-02T22:35:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 852855808 bytes 2023-12-02T22:35:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 910631936 bytes 2023-12-02T22:35:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1010693120 bytes 2023-12-02T22:35:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1068342272 bytes 2023-12-02T22:35:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1150601216 bytes 2023-12-02T22:35:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1238377472 bytes 2023-12-02T22:35:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1310040064 bytes 2023-12-02T22:35:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1337225216 bytes 2023-12-02T22:35:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1363815424 bytes 2023-12-02T22:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1364441088 bytes 2023-12-02T22:35:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1392959488 bytes 2023-12-02T22:35:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1466188800 bytes 2023-12-02T22:35:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1503045632 bytes 2023-12-02T22:35:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1503651840 bytes 2023-12-02T22:35:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1550135296 bytes 2023-12-02T22:35:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1565570048 bytes 2023-12-02T22:35:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1598287872 bytes 2023-12-02T22:35:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1652947968 bytes 2023-12-02T22:35:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1704720384 bytes 2023-12-02T22:35:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1725000704 bytes 2023-12-02T22:35:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1754708992 bytes 2023-12-02T22:35:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1809040384 bytes 2023-12-02T22:35:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1837479936 bytes 2023-12-02T22:35:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1903207424 bytes 2023-12-02T22:35:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1903809536 bytes 2023-12-02T22:35:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1910587392 bytes 2023-12-02T22:35:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1954108416 bytes 2023-12-02T22:35:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2093917184 bytes 2023-12-02T22:35:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2174188544 bytes 2023-12-02T22:35:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2253556736 bytes 2023-12-02T22:35:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2280690688 bytes 2023-12-02T22:35:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2311204864 bytes 2023-12-02T22:35:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2363814912 bytes 2023-12-02T22:35:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2440090624 bytes 2023-12-02T22:35:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2502646784 bytes 2023-12-02T22:35:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2526637056 bytes 2023-12-02T22:35:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2527263744 bytes 2023-12-02T22:35:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2608206848 bytes 2023-12-02T22:35:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2646216704 bytes 2023-12-02T22:35:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2646841344 bytes 2023-12-02T22:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2714867712 bytes 2023-12-02T22:35:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2773220352 bytes 2023-12-02T22:35:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2809417728 bytes 2023-12-02T22:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2847006720 bytes 2023-12-02T22:35:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2874468352 bytes 2023-12-02T22:35:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2926802944 bytes 2023-12-02T22:35:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2956265472 bytes 2023-12-02T22:35:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3009345536 bytes 2023-12-02T22:35:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3083198464 bytes 2023-12-02T22:35:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3108072448 bytes 2023-12-02T22:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3131045888 bytes 2023-12-02T22:35:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3143545856 bytes 2023-12-02T22:35:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3226463232 bytes 2023-12-02T22:35:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3257591808 bytes 2023-12-02T22:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3258205184 bytes 2023-12-02T22:35:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3280330752 bytes 2023-12-02T22:35:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3327630336 bytes 2023-12-02T22:35:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3339053056 bytes 2023-12-02T22:35:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3372053504 bytes 2023-12-02T22:35:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3406288896 bytes 2023-12-02T22:35:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3460555776 bytes 2023-12-02T22:35:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3515734016 bytes 2023-12-02T22:35:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3567107072 bytes 2023-12-02T22:35:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3567725568 bytes 2023-12-02T22:35:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3626352640 bytes 2023-12-02T22:35:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3675802624 bytes 2023-12-02T22:35:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3708169216 bytes 2023-12-02T22:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3747016704 bytes 2023-12-02T22:35:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3842351104 bytes 2023-12-02T22:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3864576000 bytes 2023-12-02T22:35:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3896863744 bytes 2023-12-02T22:35:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3956721664 bytes 2023-12-02T22:35:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 3995808768 bytes 2023-12-02T22:35:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4024751104 bytes 2023-12-02T22:35:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4099694592 bytes 2023-12-02T22:35:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4133788672 bytes 2023-12-02T22:35:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4172727296 bytes 2023-12-02T22:35:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4235262976 bytes 2023-12-02T22:35:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4289202176 bytes 2023-12-02T22:35:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4355142656 bytes 2023-12-02T22:35:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4408545280 bytes 2023-12-02T22:35:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4430592000 bytes 2023-12-02T22:35:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4442113024 bytes 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-02T22:35:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:35:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:35:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:35:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:35:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:35:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-02T22:35:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:35:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_15': 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 132ab2ec3f 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.132ab2ec3f 2023-12-02T22:35:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:35 2023 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:35 2023 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:35 2023 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: b7fa7b77-c7c1-4f78-b242-68962c8e8eb8 2023-12-02T22:35:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:35:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:35:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-02T22:35:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:35:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:35:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:35:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:35:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:35:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 68809728 bytes 2023-12-02T22:35:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 115380224 bytes 2023-12-02T22:35:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 119771136 bytes 2023-12-02T22:35:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 190035968 bytes 2023-12-02T22:35:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 205151232 bytes 2023-12-02T22:35:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 205766656 bytes 2023-12-02T22:35:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 290044928 bytes 2023-12-02T22:35:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 355410944 bytes 2023-12-02T22:35:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 371139584 bytes 2023-12-02T22:35:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 430118912 bytes 2023-12-02T22:35:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 483674112 bytes 2023-12-02T22:35:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 539685888 bytes 2023-12-02T22:35:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 585761792 bytes 2023-12-02T22:35:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 640636928 bytes 2023-12-02T22:35:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 643930112 bytes 2023-12-02T22:35:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 697935872 bytes 2023-12-02T22:35:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 737267712 bytes 2023-12-02T22:35:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 774284288 bytes 2023-12-02T22:35:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 790029312 bytes 2023-12-02T22:35:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 893165568 bytes 2023-12-02T22:35:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 975686656 bytes 2023-12-02T22:35:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1057179648 bytes 2023-12-02T22:35:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1101006848 bytes 2023-12-02T22:35:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1144918016 bytes 2023-12-02T22:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1221559296 bytes 2023-12-02T22:35:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1242294272 bytes 2023-12-02T22:35:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1262182400 bytes 2023-12-02T22:35:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1315459072 bytes 2023-12-02T22:35:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1388577792 bytes 2023-12-02T22:35:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1424878592 bytes 2023-12-02T22:35:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1442505728 bytes 2023-12-02T22:35:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1460901888 bytes 2023-12-02T22:35:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1464454144 bytes 2023-12-02T22:35:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1497658368 bytes 2023-12-02T22:35:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1556854784 bytes 2023-12-02T22:35:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1631213568 bytes 2023-12-02T22:35:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1695001600 bytes 2023-12-02T22:35:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1743481856 bytes 2023-12-02T22:35:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1843287040 bytes 2023-12-02T22:35:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1875468288 bytes 2023-12-02T22:35:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1888987136 bytes 2023-12-02T22:35:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1967373312 bytes 2023-12-02T22:35:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2012247040 bytes 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:35:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:35:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:35:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:35:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-12-02T22:35:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-02T22:35:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:35:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_16': 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1336f8f0af51 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1336f8f0af51 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:35:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:42 2023 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:42 2023 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:42 2023 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 99280131-88b8-4f83-930d-440a9d1456ee 2023-12-02T22:35:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:35:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:35:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-02T22:35:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:35:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:35:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:35:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:35:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:35:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 48470016 bytes 2023-12-02T22:35:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 69071872 bytes 2023-12-02T22:35:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 128096256 bytes 2023-12-02T22:35:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 158970880 bytes 2023-12-02T22:35:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 217998336 bytes 2023-12-02T22:35:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 259948544 bytes 2023-12-02T22:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 269186048 bytes 2023-12-02T22:35:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 353138688 bytes 2023-12-02T22:35:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 385902592 bytes 2023-12-02T22:35:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 405547008 bytes 2023-12-02T22:35:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 407702528 bytes 2023-12-02T22:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 492093440 bytes 2023-12-02T22:35:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 547524608 bytes 2023-12-02T22:35:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 568701952 bytes 2023-12-02T22:35:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 611949568 bytes 2023-12-02T22:35:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 632588288 bytes 2023-12-02T22:35:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 654973952 bytes 2023-12-02T22:35:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 742472704 bytes 2023-12-02T22:35:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 792458240 bytes 2023-12-02T22:35:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 803554304 bytes 2023-12-02T22:35:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 884567040 bytes 2023-12-02T22:35:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 933890048 bytes 2023-12-02T22:35:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 950366208 bytes 2023-12-02T22:35:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 976864256 bytes 2023-12-02T22:35:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1041530880 bytes 2023-12-02T22:35:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1081249792 bytes 2023-12-02T22:35:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1105410048 bytes 2023-12-02T22:35:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1173180416 bytes 2023-12-02T22:35:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1204465664 bytes 2023-12-02T22:35:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1222984704 bytes 2023-12-02T22:35:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1277351936 bytes 2023-12-02T22:35:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1315523584 bytes 2023-12-02T22:35:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1341471744 bytes 2023-12-02T22:35:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1342148608 bytes 2023-12-02T22:35:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1368309760 bytes 2023-12-02T22:35:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1427640320 bytes 2023-12-02T22:35:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1488558080 bytes 2023-12-02T22:35:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1489176576 bytes 2023-12-02T22:35:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1500378112 bytes 2023-12-02T22:35:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1548255232 bytes 2023-12-02T22:35:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1570262016 bytes 2023-12-02T22:35:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1661505536 bytes 2023-12-02T22:35:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1723602944 bytes 2023-12-02T22:35:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1786793984 bytes 2023-12-02T22:35:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1819500544 bytes 2023-12-02T22:35:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1885991936 bytes 2023-12-02T22:35:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1963276288 bytes 2023-12-02T22:35:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1974803456 bytes 2023-12-02T22:35:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2006981632 bytes 2023-12-02T22:35:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2071467008 bytes 2023-12-02T22:35:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2122021888 bytes 2023-12-02T22:35:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2158369792 bytes 2023-12-02T22:35:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2215680000 bytes 2023-12-02T22:35:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2260147200 bytes 2023-12-02T22:35:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2310044672 bytes 2023-12-02T22:35:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2364481536 bytes 2023-12-02T22:35:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2430143488 bytes 2023-12-02T22:35:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2443595776 bytes 2023-12-02T22:35:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2447880192 bytes 2023-12-02T22:35:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2482028544 bytes 2023-12-02T22:35:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2574995456 bytes 2023-12-02T22:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2591332352 bytes 2023-12-02T22:35:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2597830656 bytes 2023-12-02T22:35:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2605382656 bytes 2023-12-02T22:35:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2641403904 bytes 2023-12-02T22:35:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2698483712 bytes 2023-12-02T22:35:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2721010688 bytes 2023-12-02T22:35:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2762924032 bytes 2023-12-02T22:35:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:35:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:35:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-02T22:35:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:35:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-02T22:35:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_17': 2023-12-02T22:35:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:35:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:35:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:35:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1342b10ca4f9 2023-12-02T22:35:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1342b10ca4f9 2023-12-02T22:35:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:35:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:35:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:35:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:35:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:52 2023 2023-12-02T22:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:52 2023 2023-12-02T22:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:52 2023 2023-12-02T22:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 06d1cdd3-95e7-4459-aa25-f7cf37c99105 2023-12-02T22:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-02T22:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:35:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:35:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:35:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:35:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 64552960 bytes 2023-12-02T22:35:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 99671040 bytes 2023-12-02T22:35:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 117091328 bytes 2023-12-02T22:35:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 170612736 bytes 2023-12-02T22:35:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 196738048 bytes 2023-12-02T22:35:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 293704704 bytes 2023-12-02T22:35:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 319656960 bytes 2023-12-02T22:35:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 370546688 bytes 2023-12-02T22:35:54.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.48994877277712) > weight (0.25) 2023-12-02T22:35:54.187 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-02T22:35:54.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-02T22:35:54.187 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T22:35:54.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:35:54.186+0000 7f63a98f7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 93485) UID: 1000 2023-12-02T22:35:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 435045376 bytes 2023-12-02T22:35:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 532930560 bytes 2023-12-02T22:35:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 623248384 bytes 2023-12-02T22:35:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 623843328 bytes 2023-12-02T22:35:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 662004736 bytes 2023-12-02T22:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 672773120 bytes 2023-12-02T22:35:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 732180480 bytes 2023-12-02T22:35:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 756119552 bytes 2023-12-02T22:35:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 821939200 bytes 2023-12-02T22:35:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 859623424 bytes 2023-12-02T22:35:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 912983040 bytes 2023-12-02T22:35:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 1009100800 bytes 2023-12-02T22:35:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1050988544 bytes 2023-12-02T22:35:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1114970112 bytes 2023-12-02T22:35:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1175135232 bytes 2023-12-02T22:35:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1208462336 bytes 2023-12-02T22:35:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1218971648 bytes 2023-12-02T22:35:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1281114112 bytes 2023-12-02T22:35:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1362626560 bytes 2023-12-02T22:35:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1381321728 bytes 2023-12-02T22:35:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1415236608 bytes 2023-12-02T22:35:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1469536256 bytes 2023-12-02T22:35:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1494181888 bytes 2023-12-02T22:35:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1527680000 bytes 2023-12-02T22:35:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1545939968 bytes 2023-12-02T22:35:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1579455488 bytes 2023-12-02T22:35:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1626038272 bytes 2023-12-02T22:35:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1635261440 bytes 2023-12-02T22:35:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1705955328 bytes 2023-12-02T22:35:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1787167744 bytes 2023-12-02T22:35:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:35:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:35:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-12-02T22:35:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:35:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:35:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-02T22:35:58.188 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-02T22:35:58.188 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:35:58.189 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-02T22:35:58.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T22:35:58.189 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T22:35:58.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T22:35:58.192 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-02T22:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:35:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-02T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_18': 2023-12-02T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 134ed1dd1746 2023-12-02T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.134ed1dd1746 2023-12-02T22:35:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:35:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:35:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:35:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:35:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:58 2023 2023-12-02T22:35:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:58 2023 2023-12-02T22:35:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:58 2023 2023-12-02T22:35:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:35:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:35:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: ebdd1cd5-86de-4c0d-bc2d-c273a9a23947 2023-12-02T22:35:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:35:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:35:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-02T22:35:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:35:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:35:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:35:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:35:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2023-12-02T22:35:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:35:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:35:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 51585024 bytes 2023-12-02T22:35:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 139879424 bytes 2023-12-02T22:35:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 218696704 bytes 2023-12-02T22:35:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 256237568 bytes 2023-12-02T22:35:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 269389824 bytes 2023-12-02T22:35:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 354937856 bytes 2023-12-02T22:35:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 393927680 bytes 2023-12-02T22:35:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 429049856 bytes 2023-12-02T22:36:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 479032320 bytes 2023-12-02T22:36:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 521003008 bytes 2023-12-02T22:36:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 533353472 bytes 2023-12-02T22:36:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 555570176 bytes 2023-12-02T22:36:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 568217600 bytes 2023-12-02T22:36:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 650411008 bytes 2023-12-02T22:36:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 692925440 bytes 2023-12-02T22:36:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 747755520 bytes 2023-12-02T22:36:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 748366848 bytes 2023-12-02T22:36:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 759243776 bytes 2023-12-02T22:36:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 759846912 bytes 2023-12-02T22:36:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 770848768 bytes 2023-12-02T22:36:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 802152448 bytes 2023-12-02T22:36:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 807406592 bytes 2023-12-02T22:36:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 877007872 bytes 2023-12-02T22:36:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 891964416 bytes 2023-12-02T22:36:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 938808320 bytes 2023-12-02T22:36:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 970393600 bytes 2023-12-02T22:36:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1037285376 bytes 2023-12-02T22:36:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1082754048 bytes 2023-12-02T22:36:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1158216704 bytes 2023-12-02T22:36:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1204590592 bytes 2023-12-02T22:36:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1241967616 bytes 2023-12-02T22:36:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1262983168 bytes 2023-12-02T22:36:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1299044352 bytes 2023-12-02T22:36:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1299671040 bytes 2023-12-02T22:36:03.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7216115492200177) > weight (0.25) 2023-12-02T22:36:03.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8138825960606791) > weight (0.25) 2023-12-02T22:36:03.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4545824808837142) > weight (0.25) 2023-12-02T22:36:03.187 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-02T22:36:03.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-02T22:36:03.188 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-02T22:36:03.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:36:03.189+0000 7f11da243640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 90716) UID: 1000 2023-12-02T22:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1380415488 bytes 2023-12-02T22:36:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1381057536 bytes 2023-12-02T22:36:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1443141632 bytes 2023-12-02T22:36:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1507411968 bytes 2023-12-02T22:36:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1575921664 bytes 2023-12-02T22:36:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1652716544 bytes 2023-12-02T22:36:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1684001792 bytes 2023-12-02T22:36:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1699775488 bytes 2023-12-02T22:36:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1721256960 bytes 2023-12-02T22:36:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1825828864 bytes 2023-12-02T22:36:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1848795136 bytes 2023-12-02T22:36:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1904885760 bytes 2023-12-02T22:36:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1949936640 bytes 2023-12-02T22:36:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1990576128 bytes 2023-12-02T22:36:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2025058304 bytes 2023-12-02T22:36:05.188 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-02T22:36:05.189 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:36:05.189 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-02T22:36:05.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T22:36:05.189 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T22:36:05.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T22:36:05.192 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-02T22:36:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2067546112 bytes 2023-12-02T22:36:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2082678784 bytes 2023-12-02T22:36:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2107713536 bytes 2023-12-02T22:36:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2117400576 bytes 2023-12-02T22:36:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2122185728 bytes 2023-12-02T22:36:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2170137600 bytes 2023-12-02T22:36:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2242673664 bytes 2023-12-02T22:36:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2292363264 bytes 2023-12-02T22:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2345678848 bytes 2023-12-02T22:36:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2346279936 bytes 2023-12-02T22:36:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2361252864 bytes 2023-12-02T22:36:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2410844160 bytes 2023-12-02T22:36:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2440114176 bytes 2023-12-02T22:36:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2451483648 bytes 2023-12-02T22:36:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2558542848 bytes 2023-12-02T22:36:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2596277248 bytes 2023-12-02T22:36:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2619239424 bytes 2023-12-02T22:36:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2649221120 bytes 2023-12-02T22:36:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2704835584 bytes 2023-12-02T22:36:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2726376448 bytes 2023-12-02T22:36:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2772840448 bytes 2023-12-02T22:36:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2802212864 bytes 2023-12-02T22:36:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2846263296 bytes 2023-12-02T22:36:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2885076992 bytes 2023-12-02T22:36:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2948288512 bytes 2023-12-02T22:36:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 2953288704 bytes 2023-12-02T22:36:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3021135872 bytes 2023-12-02T22:36:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3101803520 bytes 2023-12-02T22:36:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3102442496 bytes 2023-12-02T22:36:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3164208128 bytes 2023-12-02T22:36:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3166768128 bytes 2023-12-02T22:36:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3274005504 bytes 2023-12-02T22:36:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3368364032 bytes 2023-12-02T22:36:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:36:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:36:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:36:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:36:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:36:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-02T22:36:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:36:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_19': 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1369af8dafde 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1369af8dafde 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:09 2023 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:09 2023 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:09 2023 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a8c6040f-b031-4199-aca7-736fca431324 2023-12-02T22:36:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:36:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:36:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-02T22:36:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:36:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:36:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:36:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:36:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:36:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 2577408 bytes 2023-12-02T22:36:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 11520000 bytes 2023-12-02T22:36:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 21818368 bytes 2023-12-02T22:36:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 64185344 bytes 2023-12-02T22:36:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 97472512 bytes 2023-12-02T22:36:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 192775168 bytes 2023-12-02T22:36:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 211556352 bytes 2023-12-02T22:36:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 286984192 bytes 2023-12-02T22:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 324873216 bytes 2023-12-02T22:36:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 403378176 bytes 2023-12-02T22:36:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 459559936 bytes 2023-12-02T22:36:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 480880640 bytes 2023-12-02T22:36:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 512918528 bytes 2023-12-02T22:36:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 544344064 bytes 2023-12-02T22:36:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 575281152 bytes 2023-12-02T22:36:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 605552640 bytes 2023-12-02T22:36:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 607407104 bytes 2023-12-02T22:36:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 663461888 bytes 2023-12-02T22:36:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 698089472 bytes 2023-12-02T22:36:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 707627008 bytes 2023-12-02T22:36:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 748943360 bytes 2023-12-02T22:36:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 772861952 bytes 2023-12-02T22:36:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 784036864 bytes 2023-12-02T22:36:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 814650368 bytes 2023-12-02T22:36:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 858742784 bytes 2023-12-02T22:36:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 903358464 bytes 2023-12-02T22:36:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 949654528 bytes 2023-12-02T22:36:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1014818816 bytes 2023-12-02T22:36:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1073830912 bytes 2023-12-02T22:36:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1100221440 bytes 2023-12-02T22:36:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1149108224 bytes 2023-12-02T22:36:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1270610944 bytes 2023-12-02T22:36:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1307073536 bytes 2023-12-02T22:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1381621760 bytes 2023-12-02T22:36:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1397544960 bytes 2023-12-02T22:36:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1451621376 bytes 2023-12-02T22:36:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1483428864 bytes 2023-12-02T22:36:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1523656704 bytes 2023-12-02T22:36:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1611245568 bytes 2023-12-02T22:36:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1634830336 bytes 2023-12-02T22:36:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1693470720 bytes 2023-12-02T22:36:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1738183680 bytes 2023-12-02T22:36:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1760202752 bytes 2023-12-02T22:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1792657408 bytes 2023-12-02T22:36:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1797499904 bytes 2023-12-02T22:36:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1835316224 bytes 2023-12-02T22:36:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1921629184 bytes 2023-12-02T22:36:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1962884096 bytes 2023-12-02T22:36:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2038157312 bytes 2023-12-02T22:36:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2162550784 bytes 2023-12-02T22:36:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2224935936 bytes 2023-12-02T22:36:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:36:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:36:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:36:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:36:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2023-12-02T22:36:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-02T22:36:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:36:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-12-02T22:36:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:36:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-02T22:36:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:36:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_20': 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13756f90c6aa 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13756f90c6aa 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:36:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:17 2023 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:17 2023 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:17 2023 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 89f1b9c1-adde-4973-967a-efbd5a93afd6 2023-12-02T22:36:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:36:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:36:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-02T22:36:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:36:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:36:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2023-12-02T22:36:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:36:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:36:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 21696512 bytes 2023-12-02T22:36:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 22319104 bytes 2023-12-02T22:36:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 50604032 bytes 2023-12-02T22:36:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 112988160 bytes 2023-12-02T22:36:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 124377088 bytes 2023-12-02T22:36:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 155189248 bytes 2023-12-02T22:36:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 216201216 bytes 2023-12-02T22:36:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 239111168 bytes 2023-12-02T22:36:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 276285440 bytes 2023-12-02T22:36:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 318652416 bytes 2023-12-02T22:36:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 340709376 bytes 2023-12-02T22:36:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 345819136 bytes 2023-12-02T22:36:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 403023872 bytes 2023-12-02T22:36:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 480495616 bytes 2023-12-02T22:36:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 491978752 bytes 2023-12-02T22:36:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 492596224 bytes 2023-12-02T22:36:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 529701888 bytes 2023-12-02T22:36:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 569825280 bytes 2023-12-02T22:36:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 604227584 bytes 2023-12-02T22:36:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 637243392 bytes 2023-12-02T22:36:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 674260992 bytes 2023-12-02T22:36:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 698012672 bytes 2023-12-02T22:36:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 738518016 bytes 2023-12-02T22:36:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 743867392 bytes 2023-12-02T22:36:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 786585600 bytes 2023-12-02T22:36:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 860222464 bytes 2023-12-02T22:36:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 870808576 bytes 2023-12-02T22:36:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 930104320 bytes 2023-12-02T22:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 955683840 bytes 2023-12-02T22:36:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 958885888 bytes 2023-12-02T22:36:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 988180480 bytes 2023-12-02T22:36:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1070059520 bytes 2023-12-02T22:36:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1120444416 bytes 2023-12-02T22:36:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1155236864 bytes 2023-12-02T22:36:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1254883328 bytes 2023-12-02T22:36:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1298040832 bytes 2023-12-02T22:36:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1336373248 bytes 2023-12-02T22:36:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1444212736 bytes 2023-12-02T22:36:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1502486528 bytes 2023-12-02T22:36:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1519967232 bytes 2023-12-02T22:36:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1589108736 bytes 2023-12-02T22:36:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1591567360 bytes 2023-12-02T22:36:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1647459328 bytes 2023-12-02T22:36:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1696184320 bytes 2023-12-02T22:36:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1741229056 bytes 2023-12-02T22:36:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1790020608 bytes 2023-12-02T22:36:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1818394624 bytes 2023-12-02T22:36:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1840173056 bytes 2023-12-02T22:36:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1848272896 bytes 2023-12-02T22:36:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1944124416 bytes 2023-12-02T22:36:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1958694912 bytes 2023-12-02T22:36:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2039545856 bytes 2023-12-02T22:36:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2136583168 bytes 2023-12-02T22:36:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2209210368 bytes 2023-12-02T22:36:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2232651776 bytes 2023-12-02T22:36:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2272741376 bytes 2023-12-02T22:36:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2318187520 bytes 2023-12-02T22:36:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2364024832 bytes 2023-12-02T22:36:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2444318720 bytes 2023-12-02T22:36:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2461187072 bytes 2023-12-02T22:36:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2545730560 bytes 2023-12-02T22:36:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2580285440 bytes 2023-12-02T22:36:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2698159104 bytes 2023-12-02T22:36:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2745448448 bytes 2023-12-02T22:36:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2813329408 bytes 2023-12-02T22:36:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2844551168 bytes 2023-12-02T22:36:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2904939520 bytes 2023-12-02T22:36:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2950947840 bytes 2023-12-02T22:36:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2970912768 bytes 2023-12-02T22:36:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2988463104 bytes 2023-12-02T22:36:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3063405568 bytes 2023-12-02T22:36:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3094264832 bytes 2023-12-02T22:36:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3123843072 bytes 2023-12-02T22:36:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3126471680 bytes 2023-12-02T22:36:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3201553408 bytes 2023-12-02T22:36:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3224479744 bytes 2023-12-02T22:36:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3269739520 bytes 2023-12-02T22:36:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3292349440 bytes 2023-12-02T22:36:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3311551488 bytes 2023-12-02T22:36:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3327352832 bytes 2023-12-02T22:36:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3334581248 bytes 2023-12-02T22:36:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3349157888 bytes 2023-12-02T22:36:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3385402368 bytes 2023-12-02T22:36:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3401224192 bytes 2023-12-02T22:36:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3439343616 bytes 2023-12-02T22:36:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3471311872 bytes 2023-12-02T22:36:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3524124672 bytes 2023-12-02T22:36:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3607587840 bytes 2023-12-02T22:36:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3639746560 bytes 2023-12-02T22:36:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3681591296 bytes 2023-12-02T22:36:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3733698560 bytes 2023-12-02T22:36:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3772684288 bytes 2023-12-02T22:36:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3819968512 bytes 2023-12-02T22:36:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3908469760 bytes 2023-12-02T22:36:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3940775936 bytes 2023-12-02T22:36:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4015605760 bytes 2023-12-02T22:36:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4026655744 bytes 2023-12-02T22:36:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4079356928 bytes 2023-12-02T22:36:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4144472064 bytes 2023-12-02T22:36:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4191940608 bytes 2023-12-02T22:36:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4204188672 bytes 2023-12-02T22:36:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4228537344 bytes 2023-12-02T22:36:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4280694784 bytes 2023-12-02T22:36:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4331592704 bytes 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-12-02T22:36:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:36:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:36:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-02T22:36:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:36:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-02T22:36:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:36:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_21': 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13817c175209 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13817c175209 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:32 2023 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:32 2023 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:32 2023 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 758205de-a838-479c-8d08-b70c14a4aacc 2023-12-02T22:36:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:36:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:36:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-02T22:36:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:36:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:36:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2023-12-02T22:36:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:36:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:36:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 54262784 bytes 2023-12-02T22:36:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 79122432 bytes 2023-12-02T22:36:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 103249920 bytes 2023-12-02T22:36:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 130488320 bytes 2023-12-02T22:36:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 142697472 bytes 2023-12-02T22:36:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 157163520 bytes 2023-12-02T22:36:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 184824832 bytes 2023-12-02T22:36:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 241417216 bytes 2023-12-02T22:36:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 282924032 bytes 2023-12-02T22:36:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 361678848 bytes 2023-12-02T22:36:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 395089920 bytes 2023-12-02T22:36:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 434922496 bytes 2023-12-02T22:36:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 495465472 bytes 2023-12-02T22:36:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 520053760 bytes 2023-12-02T22:36:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 560491520 bytes 2023-12-02T22:36:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 577225728 bytes 2023-12-02T22:36:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 620398592 bytes 2023-12-02T22:36:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 650342400 bytes 2023-12-02T22:36:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 697673728 bytes 2023-12-02T22:36:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 794395648 bytes 2023-12-02T22:36:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 822998016 bytes 2023-12-02T22:36:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 855543808 bytes 2023-12-02T22:36:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 886665216 bytes 2023-12-02T22:36:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 917749760 bytes 2023-12-02T22:36:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 998899712 bytes 2023-12-02T22:36:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1038941184 bytes 2023-12-02T22:36:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1058273280 bytes 2023-12-02T22:36:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1077798912 bytes 2023-12-02T22:36:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1132833792 bytes 2023-12-02T22:36:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1207220224 bytes 2023-12-02T22:36:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1260718080 bytes 2023-12-02T22:36:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1302985728 bytes 2023-12-02T22:36:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1321543680 bytes 2023-12-02T22:36:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1359594496 bytes 2023-12-02T22:36:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1440098304 bytes 2023-12-02T22:36:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1468550144 bytes 2023-12-02T22:36:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1543942144 bytes 2023-12-02T22:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1565365248 bytes 2023-12-02T22:36:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1613047808 bytes 2023-12-02T22:36:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1626116096 bytes 2023-12-02T22:36:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1655746560 bytes 2023-12-02T22:36:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1679535104 bytes 2023-12-02T22:36:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1793469440 bytes 2023-12-02T22:36:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1796639744 bytes 2023-12-02T22:36:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1821765632 bytes 2023-12-02T22:36:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1854171136 bytes 2023-12-02T22:36:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1893337088 bytes 2023-12-02T22:36:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1999229952 bytes 2023-12-02T22:36:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2077217792 bytes 2023-12-02T22:36:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2111207424 bytes 2023-12-02T22:36:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2147538944 bytes 2023-12-02T22:36:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2209584128 bytes 2023-12-02T22:36:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2307493888 bytes 2023-12-02T22:36:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2308102144 bytes 2023-12-02T22:36:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2360815616 bytes 2023-12-02T22:36:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2412602368 bytes 2023-12-02T22:36:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2435209216 bytes 2023-12-02T22:36:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2467475456 bytes 2023-12-02T22:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2484715520 bytes 2023-12-02T22:36:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2526652416 bytes 2023-12-02T22:36:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2545576960 bytes 2023-12-02T22:36:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2587016192 bytes 2023-12-02T22:36:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2587643904 bytes 2023-12-02T22:36:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2618544128 bytes 2023-12-02T22:36:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2675547136 bytes 2023-12-02T22:36:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2676163584 bytes 2023-12-02T22:36:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2696054784 bytes 2023-12-02T22:36:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2772997120 bytes 2023-12-02T22:36:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2799073280 bytes 2023-12-02T22:36:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2828968960 bytes 2023-12-02T22:36:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2884469760 bytes 2023-12-02T22:36:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2890080256 bytes 2023-12-02T22:36:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2942783488 bytes 2023-12-02T22:36:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3011034112 bytes 2023-12-02T22:36:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3042126848 bytes 2023-12-02T22:36:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3065422848 bytes 2023-12-02T22:36:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:36:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:36:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-02T22:36:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-12-02T22:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-02T22:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-02T22:36:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-02T22:36:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_22': 2023-12-02T22:36:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 138de06edbd2 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.138de06edbd2 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:43 2023 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:43 2023 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:43 2023 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:36:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 5661f1b2-001e-4d14-9686-8b906099edb6 2023-12-02T22:36:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-02T22:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:36:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:36:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:36:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:36:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 63147008 bytes 2023-12-02T22:36:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 88578048 bytes 2023-12-02T22:36:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 127759360 bytes 2023-12-02T22:36:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 137373696 bytes 2023-12-02T22:36:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 169850880 bytes 2023-12-02T22:36:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 212295680 bytes 2023-12-02T22:36:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 284801024 bytes 2023-12-02T22:36:45.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 346748928 bytes 2023-12-02T22:36:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 384957440 bytes 2023-12-02T22:36:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 427801600 bytes 2023-12-02T22:36:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 476150784 bytes 2023-12-02T22:36:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 507753472 bytes 2023-12-02T22:36:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 566165504 bytes 2023-12-02T22:36:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 611871744 bytes 2023-12-02T22:36:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 658729984 bytes 2023-12-02T22:36:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 691930112 bytes 2023-12-02T22:36:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 729905152 bytes 2023-12-02T22:36:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 810130432 bytes 2023-12-02T22:36:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 813891584 bytes 2023-12-02T22:36:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 839562240 bytes 2023-12-02T22:36:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 944555008 bytes 2023-12-02T22:36:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 945166336 bytes 2023-12-02T22:36:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1012353024 bytes 2023-12-02T22:36:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1065672704 bytes 2023-12-02T22:36:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1076208640 bytes 2023-12-02T22:36:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1091443712 bytes 2023-12-02T22:36:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1123247104 bytes 2023-12-02T22:36:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1140215808 bytes 2023-12-02T22:36:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1140845568 bytes 2023-12-02T22:36:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1248327680 bytes 2023-12-02T22:36:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1295704064 bytes 2023-12-02T22:36:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1360970752 bytes 2023-12-02T22:36:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1368511488 bytes 2023-12-02T22:36:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1424229376 bytes 2023-12-02T22:36:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1462680576 bytes 2023-12-02T22:36:49.358 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:36:49.357+0000 7f2e21bfd640 -1 reset not still connected to 0x557ceb4ab1e0 2023-12-02T22:36:49.358 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:36:49.357+0000 7f2e21bfd640 -1 reset not still connected to 0x557ceb4aba00 2023-12-02T22:36:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1506727936 bytes 2023-12-02T22:36:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1540253696 bytes 2023-12-02T22:36:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:36:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:36:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-02T22:36:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-12-02T22:36:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:36:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:36:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-02T22:36:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:36:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-02T22:36:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:36:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-02T22:36:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_23': 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1399c273af41 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1399c273af41 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:36:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:49 2023 2023-12-02T22:36:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:49 2023 2023-12-02T22:36:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:49 2023 2023-12-02T22:36:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:36:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:36:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 645a0ff2-7b25-4678-9148-2db63c48f624 2023-12-02T22:36:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:36:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:36:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-02T22:36:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:36:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:36:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:36:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:36:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:36:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 23880704 bytes 2023-12-02T22:36:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 24513536 bytes 2023-12-02T22:36:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 78770176 bytes 2023-12-02T22:36:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 190689280 bytes 2023-12-02T22:36:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 224843776 bytes 2023-12-02T22:36:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 270522368 bytes 2023-12-02T22:36:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 328080384 bytes 2023-12-02T22:36:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 366597120 bytes 2023-12-02T22:36:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 439292928 bytes 2023-12-02T22:36:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 509504512 bytes 2023-12-02T22:36:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 537993216 bytes 2023-12-02T22:36:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 554959872 bytes 2023-12-02T22:36:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 579926016 bytes 2023-12-02T22:36:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 580526080 bytes 2023-12-02T22:36:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 640056320 bytes 2023-12-02T22:36:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 720600064 bytes 2023-12-02T22:36:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 762157056 bytes 2023-12-02T22:36:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 815024128 bytes 2023-12-02T22:36:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 839141376 bytes 2023-12-02T22:36:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 882565120 bytes 2023-12-02T22:36:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 916935680 bytes 2023-12-02T22:36:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 973296640 bytes 2023-12-02T22:36:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1003674624 bytes 2023-12-02T22:36:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1025312768 bytes 2023-12-02T22:36:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1063083008 bytes 2023-12-02T22:36:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1075431424 bytes 2023-12-02T22:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1144229888 bytes 2023-12-02T22:36:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1170287616 bytes 2023-12-02T22:36:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1261532160 bytes 2023-12-02T22:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1296343040 bytes 2023-12-02T22:36:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1402839040 bytes 2023-12-02T22:36:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1509765120 bytes 2023-12-02T22:36:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1551352832 bytes 2023-12-02T22:36:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1618985984 bytes 2023-12-02T22:36:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1650556928 bytes 2023-12-02T22:36:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1710929920 bytes 2023-12-02T22:36:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1755297792 bytes 2023-12-02T22:36:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1813681152 bytes 2023-12-02T22:36:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1862691840 bytes 2023-12-02T22:36:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1890371584 bytes 2023-12-02T22:36:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1943127040 bytes 2023-12-02T22:36:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1956304896 bytes 2023-12-02T22:36:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:36:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:36:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:36:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:36:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-12-02T22:36:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:36:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:36:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-02T22:36:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:36:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_24': 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13a5a159e18e 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13a5a159e18e 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:56 2023 2023-12-02T22:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:56 2023 2023-12-02T22:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:56 2023 2023-12-02T22:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 208a8244-37c7-4cb5-8dbd-872c1e9d06f2 2023-12-02T22:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-02T22:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:36:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:36:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:36:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:36:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:36:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 26803200 bytes 2023-12-02T22:36:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 56341504 bytes 2023-12-02T22:36:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 109503488 bytes 2023-12-02T22:36:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 110113792 bytes 2023-12-02T22:36:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 116541440 bytes 2023-12-02T22:36:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 140103680 bytes 2023-12-02T22:36:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 185890816 bytes 2023-12-02T22:36:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 248795136 bytes 2023-12-02T22:36:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 291732480 bytes 2023-12-02T22:36:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 317789184 bytes 2023-12-02T22:36:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 357756928 bytes 2023-12-02T22:36:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 427417600 bytes 2023-12-02T22:36:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 442511360 bytes 2023-12-02T22:36:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 502335488 bytes 2023-12-02T22:36:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 504918016 bytes 2023-12-02T22:36:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 507772928 bytes 2023-12-02T22:36:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 539993088 bytes 2023-12-02T22:36:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 567177216 bytes 2023-12-02T22:37:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 592959488 bytes 2023-12-02T22:37:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 617293824 bytes 2023-12-02T22:37:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 699027456 bytes 2023-12-02T22:37:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 699654144 bytes 2023-12-02T22:37:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 734699520 bytes 2023-12-02T22:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 745100288 bytes 2023-12-02T22:37:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 800544768 bytes 2023-12-02T22:37:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 831581184 bytes 2023-12-02T22:37:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 869596160 bytes 2023-12-02T22:37:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 939985920 bytes 2023-12-02T22:37:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 957886464 bytes 2023-12-02T22:37:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1049279488 bytes 2023-12-02T22:37:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1099590656 bytes 2023-12-02T22:37:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1157605376 bytes 2023-12-02T22:37:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1177483264 bytes 2023-12-02T22:37:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1197282304 bytes 2023-12-02T22:37:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1269892096 bytes 2023-12-02T22:37:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1300484096 bytes 2023-12-02T22:37:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1371402240 bytes 2023-12-02T22:37:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1388538880 bytes 2023-12-02T22:37:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:37:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:37:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-12-02T22:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-02T22:37:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:37:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-02T22:37:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:37:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-02T22:37:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_25': 2023-12-02T22:37:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:37:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13b1dc62da4d 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13b1dc62da4d 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:02 2023 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:02 2023 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:02 2023 2023-12-02T22:37:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:37:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:37:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c16dc03c-c292-424e-be08-d6e908fda7e8 2023-12-02T22:37:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:37:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:37:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-02T22:37:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:37:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:37:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2023-12-02T22:37:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:37:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:37:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 25006080 bytes 2023-12-02T22:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 85522432 bytes 2023-12-02T22:37:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 104600576 bytes 2023-12-02T22:37:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 114768896 bytes 2023-12-02T22:37:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 198297600 bytes 2023-12-02T22:37:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 245138432 bytes 2023-12-02T22:37:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 261588992 bytes 2023-12-02T22:37:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 293593088 bytes 2023-12-02T22:37:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 345189376 bytes 2023-12-02T22:37:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 356765696 bytes 2023-12-02T22:37:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 389544960 bytes 2023-12-02T22:37:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 437879808 bytes 2023-12-02T22:37:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 461324288 bytes 2023-12-02T22:37:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 513897472 bytes 2023-12-02T22:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 538110976 bytes 2023-12-02T22:37:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 553510912 bytes 2023-12-02T22:37:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 603863040 bytes 2023-12-02T22:37:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 656622592 bytes 2023-12-02T22:37:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 682513408 bytes 2023-12-02T22:37:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 726586368 bytes 2023-12-02T22:37:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 783610880 bytes 2023-12-02T22:37:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 806156288 bytes 2023-12-02T22:37:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 902484992 bytes 2023-12-02T22:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 909770752 bytes 2023-12-02T22:37:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 941451264 bytes 2023-12-02T22:37:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1000619008 bytes 2023-12-02T22:37:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1068043264 bytes 2023-12-02T22:37:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1093673984 bytes 2023-12-02T22:37:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1138504704 bytes 2023-12-02T22:37:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1160200192 bytes 2023-12-02T22:37:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1176498176 bytes 2023-12-02T22:37:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1202740224 bytes 2023-12-02T22:37:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1251251200 bytes 2023-12-02T22:37:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1275653120 bytes 2023-12-02T22:37:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1287999488 bytes 2023-12-02T22:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1310291968 bytes 2023-12-02T22:37:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1380796416 bytes 2023-12-02T22:37:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1413601280 bytes 2023-12-02T22:37:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1481023488 bytes 2023-12-02T22:37:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1511479296 bytes 2023-12-02T22:37:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1542865920 bytes 2023-12-02T22:37:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1596698624 bytes 2023-12-02T22:37:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1634530304 bytes 2023-12-02T22:37:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1659153408 bytes 2023-12-02T22:37:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1717518336 bytes 2023-12-02T22:37:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1797090304 bytes 2023-12-02T22:37:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1817049088 bytes 2023-12-02T22:37:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1863266304 bytes 2023-12-02T22:37:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1893787648 bytes 2023-12-02T22:37:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1918920704 bytes 2023-12-02T22:37:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1982686208 bytes 2023-12-02T22:37:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2020519936 bytes 2023-12-02T22:37:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2078070784 bytes 2023-12-02T22:37:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2116587520 bytes 2023-12-02T22:37:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2168083456 bytes 2023-12-02T22:37:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2235941888 bytes 2023-12-02T22:37:11.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9127832925999839) > weight (0.25) 2023-12-02T22:37:11.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6940531242177793) > weight (0.25) 2023-12-02T22:37:11.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3036876903930259) > weight (0.25) 2023-12-02T22:37:11.194 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-02T22:37:11.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-02T22:37:11.195 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-02T22:37:11.195 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-02T22:37:11.195 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:37:11.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:37:11.193+0000 7f7051729640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 100827) UID: 1000 2023-12-02T22:37:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2335011840 bytes 2023-12-02T22:37:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2350586880 bytes 2023-12-02T22:37:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2376736768 bytes 2023-12-02T22:37:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2438279168 bytes 2023-12-02T22:37:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2469652480 bytes 2023-12-02T22:37:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2499651584 bytes 2023-12-02T22:37:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2556657664 bytes 2023-12-02T22:37:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2574430208 bytes 2023-12-02T22:37:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2601417728 bytes 2023-12-02T22:37:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2645049344 bytes 2023-12-02T22:37:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2665472000 bytes 2023-12-02T22:37:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2698417152 bytes 2023-12-02T22:37:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2731923456 bytes 2023-12-02T22:37:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2749038592 bytes 2023-12-02T22:37:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2774608896 bytes 2023-12-02T22:37:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2834883584 bytes 2023-12-02T22:37:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2840146944 bytes 2023-12-02T22:37:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2886703104 bytes 2023-12-02T22:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 2932111360 bytes 2023-12-02T22:37:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 2976801792 bytes 2023-12-02T22:37:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 2977401856 bytes 2023-12-02T22:37:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3019623424 bytes 2023-12-02T22:37:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3077327872 bytes 2023-12-02T22:37:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3142520832 bytes 2023-12-02T22:37:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3196190720 bytes 2023-12-02T22:37:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3221857280 bytes 2023-12-02T22:37:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3230611456 bytes 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:37:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-02T22:37:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-02T22:37:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:37:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-02T22:37:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_26': 2023-12-02T22:37:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13bd6b3e9b43 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13bd6b3e9b43 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:14 2023 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:14 2023 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:14 2023 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 315680d7-c2d3-4042-8d33-632d758e85f2 2023-12-02T22:37:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:37:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:37:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-02T22:37:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:37:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:37:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:37:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:37:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:37:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 48152576 bytes 2023-12-02T22:37:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 115047424 bytes 2023-12-02T22:37:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 129354752 bytes 2023-12-02T22:37:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 188580864 bytes 2023-12-02T22:37:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 227612672 bytes 2023-12-02T22:37:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 285004800 bytes 2023-12-02T22:37:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 285616128 bytes 2023-12-02T22:37:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 327832576 bytes 2023-12-02T22:37:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 359748608 bytes 2023-12-02T22:37:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 375926784 bytes 2023-12-02T22:37:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 446158848 bytes 2023-12-02T22:37:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 451937280 bytes 2023-12-02T22:37:17.296 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-02T22:37:17.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T22:37:17.297 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T22:37:17.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T22:37:17.301 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-02T22:37:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 575433728 bytes 2023-12-02T22:37:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 611420160 bytes 2023-12-02T22:37:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 664819712 bytes 2023-12-02T22:37:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 665426944 bytes 2023-12-02T22:37:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 725859328 bytes 2023-12-02T22:37:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 832645120 bytes 2023-12-02T22:37:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 861706240 bytes 2023-12-02T22:37:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 910500864 bytes 2023-12-02T22:37:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 967192576 bytes 2023-12-02T22:37:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 994041856 bytes 2023-12-02T22:37:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1023073280 bytes 2023-12-02T22:37:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1058846720 bytes 2023-12-02T22:37:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1093629952 bytes 2023-12-02T22:37:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1130636288 bytes 2023-12-02T22:37:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1131219968 bytes 2023-12-02T22:37:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1131828224 bytes 2023-12-02T22:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1166747648 bytes 2023-12-02T22:37:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1212348416 bytes 2023-12-02T22:37:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1285372928 bytes 2023-12-02T22:37:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1334789120 bytes 2023-12-02T22:37:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1345783808 bytes 2023-12-02T22:37:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1363911680 bytes 2023-12-02T22:37:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1407307776 bytes 2023-12-02T22:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1503295488 bytes 2023-12-02T22:37:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1589561344 bytes 2023-12-02T22:37:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1599794176 bytes 2023-12-02T22:37:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1645512704 bytes 2023-12-02T22:37:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1649012736 bytes 2023-12-02T22:37:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1659611136 bytes 2023-12-02T22:37:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1695895552 bytes 2023-12-02T22:37:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1727981568 bytes 2023-12-02T22:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1759994880 bytes 2023-12-02T22:37:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1801253888 bytes 2023-12-02T22:37:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1850295296 bytes 2023-12-02T22:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1949580288 bytes 2023-12-02T22:37:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2046686208 bytes 2023-12-02T22:37:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2120810496 bytes 2023-12-02T22:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2128822272 bytes 2023-12-02T22:37:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2179542016 bytes 2023-12-02T22:37:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2290799616 bytes 2023-12-02T22:37:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2301715456 bytes 2023-12-02T22:37:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2354591744 bytes 2023-12-02T22:37:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2368367616 bytes 2023-12-02T22:37:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2414572544 bytes 2023-12-02T22:37:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2484268032 bytes 2023-12-02T22:37:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:37:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:37:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-12-02T22:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-02T22:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-02T22:37:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:37:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-02T22:37:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_27': 2023-12-02T22:37:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:37:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:37:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:37:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13cdeccd63d 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13cdeccd63d 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:23 2023 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:23 2023 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:23 2023 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: cba1b5db-1e9a-4652-8e1d-956d8a38b3c8 2023-12-02T22:37:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:37:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:37:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-02T22:37:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:37:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:37:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:37:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:37:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:37:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 616448 bytes 2023-12-02T22:37:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 6718464 bytes 2023-12-02T22:37:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 77917184 bytes 2023-12-02T22:37:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 118639616 bytes 2023-12-02T22:37:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 163969024 bytes 2023-12-02T22:37:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 242740224 bytes 2023-12-02T22:37:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 284750848 bytes 2023-12-02T22:37:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 302981120 bytes 2023-12-02T22:37:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 330277888 bytes 2023-12-02T22:37:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 379640832 bytes 2023-12-02T22:37:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 383685632 bytes 2023-12-02T22:37:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 401112064 bytes 2023-12-02T22:37:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 433582080 bytes 2023-12-02T22:37:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 499117056 bytes 2023-12-02T22:37:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 514568192 bytes 2023-12-02T22:37:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 653069312 bytes 2023-12-02T22:37:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 653691904 bytes 2023-12-02T22:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 654307328 bytes 2023-12-02T22:37:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 664564736 bytes 2023-12-02T22:37:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 734721024 bytes 2023-12-02T22:37:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 735342592 bytes 2023-12-02T22:37:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 804177920 bytes 2023-12-02T22:37:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 813648896 bytes 2023-12-02T22:37:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 816480256 bytes 2023-12-02T22:37:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 835622912 bytes 2023-12-02T22:37:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 863548416 bytes 2023-12-02T22:37:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 898748416 bytes 2023-12-02T22:37:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 927955968 bytes 2023-12-02T22:37:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 936452096 bytes 2023-12-02T22:37:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 971760640 bytes 2023-12-02T22:37:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1003079680 bytes 2023-12-02T22:37:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1020453888 bytes 2023-12-02T22:37:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1046194176 bytes 2023-12-02T22:37:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1055178752 bytes 2023-12-02T22:37:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1080198144 bytes 2023-12-02T22:37:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1114525696 bytes 2023-12-02T22:37:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1179948032 bytes 2023-12-02T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1213433856 bytes 2023-12-02T22:37:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1242223616 bytes 2023-12-02T22:37:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1290548224 bytes 2023-12-02T22:37:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1347798016 bytes 2023-12-02T22:37:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1374490624 bytes 2023-12-02T22:37:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1410531328 bytes 2023-12-02T22:37:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1474867200 bytes 2023-12-02T22:37:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1520425984 bytes 2023-12-02T22:37:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1583804416 bytes 2023-12-02T22:37:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1605740544 bytes 2023-12-02T22:37:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1608873984 bytes 2023-12-02T22:37:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1660153856 bytes 2023-12-02T22:37:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1719742464 bytes 2023-12-02T22:37:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1810523136 bytes 2023-12-02T22:37:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1823644672 bytes 2023-12-02T22:37:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1839123456 bytes 2023-12-02T22:37:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:37:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:37:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:37:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-02T22:37:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:37:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_28': 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13d6c2f06848 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13d6c2f06848 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:37:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:31 2023 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:31 2023 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:31 2023 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 65974775-a20e-4c7a-9d15-5a68ebf338c6 2023-12-02T22:37:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:37:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:37:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-02T22:37:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:37:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:37:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2023-12-02T22:37:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:37:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 64064512 bytes 2023-12-02T22:37:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 87407616 bytes 2023-12-02T22:37:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 162159616 bytes 2023-12-02T22:37:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 185379840 bytes 2023-12-02T22:37:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 194825216 bytes 2023-12-02T22:37:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 245608448 bytes 2023-12-02T22:37:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 270653440 bytes 2023-12-02T22:37:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 280546304 bytes 2023-12-02T22:37:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 383629312 bytes 2023-12-02T22:37:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 409390080 bytes 2023-12-02T22:37:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 421977088 bytes 2023-12-02T22:37:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 438021120 bytes 2023-12-02T22:37:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 475689984 bytes 2023-12-02T22:37:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 505023488 bytes 2023-12-02T22:37:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 524743680 bytes 2023-12-02T22:37:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 577951744 bytes 2023-12-02T22:37:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 605576192 bytes 2023-12-02T22:37:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 618199040 bytes 2023-12-02T22:37:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 666318848 bytes 2023-12-02T22:37:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 731555840 bytes 2023-12-02T22:37:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 746930176 bytes 2023-12-02T22:37:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 765412352 bytes 2023-12-02T22:37:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 775032832 bytes 2023-12-02T22:37:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 816409600 bytes 2023-12-02T22:37:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 907447296 bytes 2023-12-02T22:37:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 999191552 bytes 2023-12-02T22:37:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1108201472 bytes 2023-12-02T22:37:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1176757248 bytes 2023-12-02T22:37:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1224600576 bytes 2023-12-02T22:37:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1256356864 bytes 2023-12-02T22:37:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1305926656 bytes 2023-12-02T22:37:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1344142336 bytes 2023-12-02T22:37:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1404524544 bytes 2023-12-02T22:37:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1440843776 bytes 2023-12-02T22:37:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1510538240 bytes 2023-12-02T22:37:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1549760512 bytes 2023-12-02T22:37:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1598600192 bytes 2023-12-02T22:37:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1601338368 bytes 2023-12-02T22:37:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1666178048 bytes 2023-12-02T22:37:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1666791424 bytes 2023-12-02T22:37:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1700203520 bytes 2023-12-02T22:37:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1719628800 bytes 2023-12-02T22:37:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1775510528 bytes 2023-12-02T22:37:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1842242560 bytes 2023-12-02T22:37:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1930744832 bytes 2023-12-02T22:37:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1988683776 bytes 2023-12-02T22:37:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2037678080 bytes 2023-12-02T22:37:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2038300672 bytes 2023-12-02T22:37:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2192668672 bytes 2023-12-02T22:37:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2230834176 bytes 2023-12-02T22:37:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2290445312 bytes 2023-12-02T22:37:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2306102272 bytes 2023-12-02T22:37:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2350714880 bytes 2023-12-02T22:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2402656256 bytes 2023-12-02T22:37:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2434415616 bytes 2023-12-02T22:37:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2480708608 bytes 2023-12-02T22:37:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2531807232 bytes 2023-12-02T22:37:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2576557056 bytes 2023-12-02T22:37:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2618070016 bytes 2023-12-02T22:37:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2688808960 bytes 2023-12-02T22:37:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2711122944 bytes 2023-12-02T22:37:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2797969408 bytes 2023-12-02T22:37:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2839218176 bytes 2023-12-02T22:37:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2906248192 bytes 2023-12-02T22:37:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2961444864 bytes 2023-12-02T22:37:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2987278336 bytes 2023-12-02T22:37:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3031580672 bytes 2023-12-02T22:37:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3071469568 bytes 2023-12-02T22:37:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3122745344 bytes 2023-12-02T22:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3198712832 bytes 2023-12-02T22:37:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3237770240 bytes 2023-12-02T22:37:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3250433024 bytes 2023-12-02T22:37:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3399453696 bytes 2023-12-02T22:37:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3440225280 bytes 2023-12-02T22:37:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3440875520 bytes 2023-12-02T22:37:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3538813952 bytes 2023-12-02T22:37:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3561381888 bytes 2023-12-02T22:37:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3562035200 bytes 2023-12-02T22:37:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3598504960 bytes 2023-12-02T22:37:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3666076672 bytes 2023-12-02T22:37:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3724905472 bytes 2023-12-02T22:37:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3762064384 bytes 2023-12-02T22:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3825779712 bytes 2023-12-02T22:37:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3868889088 bytes 2023-12-02T22:37:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3924012032 bytes 2023-12-02T22:37:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3966406656 bytes 2023-12-02T22:37:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3980971008 bytes 2023-12-02T22:37:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3984769024 bytes 2023-12-02T22:37:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 4019757056 bytes 2023-12-02T22:37:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 4054369280 bytes 2023-12-02T22:37:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 4054979584 bytes 2023-12-02T22:37:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 4100977664 bytes 2023-12-02T22:37:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 4125371392 bytes 2023-12-02T22:37:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 4156034048 bytes 2023-12-02T22:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 4192471040 bytes 2023-12-02T22:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4201023488 bytes 2023-12-02T22:37:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4220711936 bytes 2023-12-02T22:37:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4309473280 bytes 2023-12-02T22:37:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4344258560 bytes 2023-12-02T22:37:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4394734592 bytes 2023-12-02T22:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-12-02T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-02T22:37:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:37:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_29': 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13de5063ffe 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13de5063ffe 2023-12-02T22:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:46 2023 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:46 2023 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:46 2023 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8686cb37-c0f3-4cb1-bfb7-7d89f4ef29c2 2023-12-02T22:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-02T22:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:37:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:37:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:37:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:37:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 80540672 bytes 2023-12-02T22:37:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 153678848 bytes 2023-12-02T22:37:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 181664768 bytes 2023-12-02T22:37:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 182292480 bytes 2023-12-02T22:37:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 251665408 bytes 2023-12-02T22:37:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 252259328 bytes 2023-12-02T22:37:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 297612288 bytes 2023-12-02T22:37:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 357000192 bytes 2023-12-02T22:37:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 371572736 bytes 2023-12-02T22:37:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 424201216 bytes 2023-12-02T22:37:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 465289216 bytes 2023-12-02T22:37:48.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9488392241019106) > weight (0.25) 2023-12-02T22:37:48.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9894929651722691) > weight (0.25) 2023-12-02T22:37:48.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6299669399162383) > weight (0.25) 2023-12-02T22:37:48.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7506582106099877) > weight (0.25) 2023-12-02T22:37:48.193 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-02T22:37:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 484071424 bytes 2023-12-02T22:37:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 530102272 bytes 2023-12-02T22:37:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 594450432 bytes 2023-12-02T22:37:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 629170176 bytes 2023-12-02T22:37:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 676769792 bytes 2023-12-02T22:37:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 698105856 bytes 2023-12-02T22:37:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 732007424 bytes 2023-12-02T22:37:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 797104128 bytes 2023-12-02T22:37:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 843860992 bytes 2023-12-02T22:37:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 959473664 bytes 2023-12-02T22:37:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1031080960 bytes 2023-12-02T22:37:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1046297600 bytes 2023-12-02T22:37:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1056700416 bytes 2023-12-02T22:37:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1087122432 bytes 2023-12-02T22:37:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1130764288 bytes 2023-12-02T22:37:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1233021952 bytes 2023-12-02T22:37:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1266262016 bytes 2023-12-02T22:37:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1326152704 bytes 2023-12-02T22:37:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1378963456 bytes 2023-12-02T22:37:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1440161792 bytes 2023-12-02T22:37:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1462702080 bytes 2023-12-02T22:37:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1467557888 bytes 2023-12-02T22:37:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1559212032 bytes 2023-12-02T22:37:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1590993920 bytes 2023-12-02T22:37:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1670414336 bytes 2023-12-02T22:37:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1699593216 bytes 2023-12-02T22:37:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1726480384 bytes 2023-12-02T22:37:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1787766784 bytes 2023-12-02T22:37:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1847159808 bytes 2023-12-02T22:37:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1912368128 bytes 2023-12-02T22:37:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1956929536 bytes 2023-12-02T22:37:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2039281664 bytes 2023-12-02T22:37:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2062877696 bytes 2023-12-02T22:37:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2080437248 bytes 2023-12-02T22:37:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2136434688 bytes 2023-12-02T22:37:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2148022272 bytes 2023-12-02T22:37:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2206982144 bytes 2023-12-02T22:37:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2253240320 bytes 2023-12-02T22:37:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2283607040 bytes 2023-12-02T22:37:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:37:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:37:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:37:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-02T22:37:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:37:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-12-02T22:37:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:37:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-02T22:37:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:37:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-02T22:37:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_30': 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13e6f41b6980 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13e6f41b6980 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:37:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:53 2023 2023-12-02T22:37:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:53 2023 2023-12-02T22:37:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:53 2023 2023-12-02T22:37:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:37:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:37:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: af5e53fb-d079-4e24-8f20-85b325a485b5 2023-12-02T22:37:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:37:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:37:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-02T22:37:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:37:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:37:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:37:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:37:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:37:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:37:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:37:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 79193088 bytes 2023-12-02T22:37:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 128328704 bytes 2023-12-02T22:37:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 140183552 bytes 2023-12-02T22:37:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 146950144 bytes 2023-12-02T22:37:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 171415552 bytes 2023-12-02T22:37:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 231234560 bytes 2023-12-02T22:37:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 257533952 bytes 2023-12-02T22:37:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 262681600 bytes 2023-12-02T22:37:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 332545024 bytes 2023-12-02T22:37:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 390132736 bytes 2023-12-02T22:37:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 420496384 bytes 2023-12-02T22:37:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 442437632 bytes 2023-12-02T22:37:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 462252032 bytes 2023-12-02T22:37:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 565332992 bytes 2023-12-02T22:37:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 606330880 bytes 2023-12-02T22:37:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 653379584 bytes 2023-12-02T22:37:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 698534912 bytes 2023-12-02T22:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 726477824 bytes 2023-12-02T22:37:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 727075840 bytes 2023-12-02T22:37:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 770049024 bytes 2023-12-02T22:37:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 805099520 bytes 2023-12-02T22:37:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 859385856 bytes 2023-12-02T22:37:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 908726272 bytes 2023-12-02T22:37:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 954089472 bytes 2023-12-02T22:37:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1026421760 bytes 2023-12-02T22:37:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1064275968 bytes 2023-12-02T22:37:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1110559744 bytes 2023-12-02T22:37:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1115617280 bytes 2023-12-02T22:37:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1147540480 bytes 2023-12-02T22:37:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1232425984 bytes 2023-12-02T22:37:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1267667968 bytes 2023-12-02T22:37:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1336499200 bytes 2023-12-02T22:37:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1356305408 bytes 2023-12-02T22:37:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1382714368 bytes 2023-12-02T22:37:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1402906624 bytes 2023-12-02T22:37:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1468906496 bytes 2023-12-02T22:37:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1500597248 bytes 2023-12-02T22:38:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1501235200 bytes 2023-12-02T22:38:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1541326848 bytes 2023-12-02T22:38:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1583509504 bytes 2023-12-02T22:38:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1650394112 bytes 2023-12-02T22:38:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1699667968 bytes 2023-12-02T22:38:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1737674752 bytes 2023-12-02T22:38:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1786209280 bytes 2023-12-02T22:38:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1798657024 bytes 2023-12-02T22:38:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1849665536 bytes 2023-12-02T22:38:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1866680320 bytes 2023-12-02T22:38:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1884841984 bytes 2023-12-02T22:38:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1917959168 bytes 2023-12-02T22:38:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2025282560 bytes 2023-12-02T22:38:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2088749056 bytes 2023-12-02T22:38:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2151726080 bytes 2023-12-02T22:38:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2195033088 bytes 2023-12-02T22:38:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2241368064 bytes 2023-12-02T22:38:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2315273216 bytes 2023-12-02T22:38:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-02T22:38:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-12-02T22:38:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-02T22:38:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:38:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-02T22:38:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_31': 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13ee9c8f7fcf 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13ee9c8f7fcf 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:02 2023 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:02 2023 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:02 2023 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 93fcda49-830e-47ed-9f3a-10f4ff483bab 2023-12-02T22:38:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-02T22:38:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:38:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:38:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 61649920 bytes 2023-12-02T22:38:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 62290944 bytes 2023-12-02T22:38:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 73154560 bytes 2023-12-02T22:38:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 101016576 bytes 2023-12-02T22:38:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 101616640 bytes 2023-12-02T22:38:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 170193920 bytes 2023-12-02T22:38:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 214529024 bytes 2023-12-02T22:38:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 266010624 bytes 2023-12-02T22:38:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 283835392 bytes 2023-12-02T22:38:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 339325952 bytes 2023-12-02T22:38:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 381080576 bytes 2023-12-02T22:38:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 401576960 bytes 2023-12-02T22:38:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 429488128 bytes 2023-12-02T22:38:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 465340416 bytes 2023-12-02T22:38:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 534752256 bytes 2023-12-02T22:38:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 612666368 bytes 2023-12-02T22:38:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 681650176 bytes 2023-12-02T22:38:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 708806656 bytes 2023-12-02T22:38:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 762333184 bytes 2023-12-02T22:38:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 804421632 bytes 2023-12-02T22:38:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 840635392 bytes 2023-12-02T22:38:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 932497408 bytes 2023-12-02T22:38:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 959143936 bytes 2023-12-02T22:38:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1055783936 bytes 2023-12-02T22:38:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1068830720 bytes 2023-12-02T22:38:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1123790848 bytes 2023-12-02T22:38:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1154306048 bytes 2023-12-02T22:38:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1263571968 bytes 2023-12-02T22:38:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1279662080 bytes 2023-12-02T22:38:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1297401856 bytes 2023-12-02T22:38:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1362440192 bytes 2023-12-02T22:38:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1407474688 bytes 2023-12-02T22:38:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1459206144 bytes 2023-12-02T22:38:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1509978112 bytes 2023-12-02T22:38:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1546221568 bytes 2023-12-02T22:38:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1588556800 bytes 2023-12-02T22:38:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-12-02T22:38:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-02T22:38:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:38:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-02T22:38:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-02T22:38:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_32': 2023-12-02T22:38:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 13f823238366 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.13f823238366 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:08 2023 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:08 2023 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:08 2023 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 662b719c-5386-4ec4-8056-aa4e67a37e59 2023-12-02T22:38:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-02T22:38:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:38:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:38:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 78253056 bytes 2023-12-02T22:38:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 102285312 bytes 2023-12-02T22:38:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 155473920 bytes 2023-12-02T22:38:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 204065792 bytes 2023-12-02T22:38:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 253835264 bytes 2023-12-02T22:38:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 305681408 bytes 2023-12-02T22:38:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 313595904 bytes 2023-12-02T22:38:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 353207296 bytes 2023-12-02T22:38:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 392572928 bytes 2023-12-02T22:38:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 402791424 bytes 2023-12-02T22:38:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 451123200 bytes 2023-12-02T22:38:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 500703232 bytes 2023-12-02T22:38:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 533342208 bytes 2023-12-02T22:38:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 597421056 bytes 2023-12-02T22:38:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 624913408 bytes 2023-12-02T22:38:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 658966528 bytes 2023-12-02T22:38:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 669722624 bytes 2023-12-02T22:38:12.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 696101888 bytes 2023-12-02T22:38:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 737308672 bytes 2023-12-02T22:38:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 781884416 bytes 2023-12-02T22:38:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 801988608 bytes 2023-12-02T22:38:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 804964352 bytes 2023-12-02T22:38:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 822020096 bytes 2023-12-02T22:38:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 837117952 bytes 2023-12-02T22:38:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 852888576 bytes 2023-12-02T22:38:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 898521088 bytes 2023-12-02T22:38:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 948654080 bytes 2023-12-02T22:38:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 978537472 bytes 2023-12-02T22:38:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 982182912 bytes 2023-12-02T22:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1048262656 bytes 2023-12-02T22:38:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1088529408 bytes 2023-12-02T22:38:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1131446272 bytes 2023-12-02T22:38:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1159500800 bytes 2023-12-02T22:38:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1205624832 bytes 2023-12-02T22:38:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1228375040 bytes 2023-12-02T22:38:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1268617216 bytes 2023-12-02T22:38:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1276983296 bytes 2023-12-02T22:38:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1328370688 bytes 2023-12-02T22:38:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1356789760 bytes 2023-12-02T22:38:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1404639232 bytes 2023-12-02T22:38:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1432216576 bytes 2023-12-02T22:38:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1469049856 bytes 2023-12-02T22:38:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1529097216 bytes 2023-12-02T22:38:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1559050240 bytes 2023-12-02T22:38:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1631975424 bytes 2023-12-02T22:38:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1666240512 bytes 2023-12-02T22:38:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1752113152 bytes 2023-12-02T22:38:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1805299712 bytes 2023-12-02T22:38:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1813013504 bytes 2023-12-02T22:38:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1834919936 bytes 2023-12-02T22:38:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1899769856 bytes 2023-12-02T22:38:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1912145920 bytes 2023-12-02T22:38:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1931302912 bytes 2023-12-02T22:38:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 1961100288 bytes 2023-12-02T22:38:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 1985626112 bytes 2023-12-02T22:38:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2014195712 bytes 2023-12-02T22:38:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2097353728 bytes 2023-12-02T22:38:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2105716736 bytes 2023-12-02T22:38:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2171683840 bytes 2023-12-02T22:38:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2206503936 bytes 2023-12-02T22:38:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2247282688 bytes 2023-12-02T22:38:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2322173952 bytes 2023-12-02T22:38:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2380390400 bytes 2023-12-02T22:38:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2435887104 bytes 2023-12-02T22:38:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2463558656 bytes 2023-12-02T22:38:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2531701760 bytes 2023-12-02T22:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2589323264 bytes 2023-12-02T22:38:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2653173760 bytes 2023-12-02T22:38:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2672519168 bytes 2023-12-02T22:38:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2723466240 bytes 2023-12-02T22:38:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2740824064 bytes 2023-12-02T22:38:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2796002304 bytes 2023-12-02T22:38:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2826561536 bytes 2023-12-02T22:38:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2846223360 bytes 2023-12-02T22:38:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 2912405504 bytes 2023-12-02T22:38:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 2967761920 bytes 2023-12-02T22:38:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3010279424 bytes 2023-12-02T22:38:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3029949440 bytes 2023-12-02T22:38:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3030563840 bytes 2023-12-02T22:38:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3054101504 bytes 2023-12-02T22:38:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3097414656 bytes 2023-12-02T22:38:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3195047936 bytes 2023-12-02T22:38:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3228770304 bytes 2023-12-02T22:38:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3279659008 bytes 2023-12-02T22:38:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3342513152 bytes 2023-12-02T22:38:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3395365888 bytes 2023-12-02T22:38:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3407401984 bytes 2023-12-02T22:38:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3498937344 bytes 2023-12-02T22:38:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3522509824 bytes 2023-12-02T22:38:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-02T22:38:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-02T22:38:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_33': 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1401656dc49d 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1401656dc49d 2023-12-02T22:38:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:21 2023 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:21 2023 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:21 2023 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d64bed83-0a8b-46ea-b180-1169dba0dac7 2023-12-02T22:38:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-02T22:38:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:38:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:38:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 36418560 bytes 2023-12-02T22:38:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 64322560 bytes 2023-12-02T22:38:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 101342208 bytes 2023-12-02T22:38:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 128869376 bytes 2023-12-02T22:38:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 173641728 bytes 2023-12-02T22:38:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 204121088 bytes 2023-12-02T22:38:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 204739584 bytes 2023-12-02T22:38:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 255429632 bytes 2023-12-02T22:38:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 314048512 bytes 2023-12-02T22:38:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 399367168 bytes 2023-12-02T22:38:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 421893120 bytes 2023-12-02T22:38:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 462007296 bytes 2023-12-02T22:38:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 480781312 bytes 2023-12-02T22:38:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 577528832 bytes 2023-12-02T22:38:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 646456320 bytes 2023-12-02T22:38:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 688261120 bytes 2023-12-02T22:38:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 691154944 bytes 2023-12-02T22:38:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 754451456 bytes 2023-12-02T22:38:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 812399616 bytes 2023-12-02T22:38:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 841262080 bytes 2023-12-02T22:38:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 899690496 bytes 2023-12-02T22:38:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 917776384 bytes 2023-12-02T22:38:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 938858496 bytes 2023-12-02T22:38:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1035996160 bytes 2023-12-02T22:38:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1058046976 bytes 2023-12-02T22:38:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1106217984 bytes 2023-12-02T22:38:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1167421440 bytes 2023-12-02T22:38:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1174368256 bytes 2023-12-02T22:38:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1201920000 bytes 2023-12-02T22:38:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1226041344 bytes 2023-12-02T22:38:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1246449664 bytes 2023-12-02T22:38:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1315129344 bytes 2023-12-02T22:38:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1508336640 bytes 2023-12-02T22:38:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1516135424 bytes 2023-12-02T22:38:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1583559680 bytes 2023-12-02T22:38:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1623401472 bytes 2023-12-02T22:38:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1717104640 bytes 2023-12-02T22:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1735267328 bytes 2023-12-02T22:38:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1783702528 bytes 2023-12-02T22:38:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1855594496 bytes 2023-12-02T22:38:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1903364096 bytes 2023-12-02T22:38:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1912437760 bytes 2023-12-02T22:38:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1913042944 bytes 2023-12-02T22:38:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1939019776 bytes 2023-12-02T22:38:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1949745152 bytes 2023-12-02T22:38:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1965837312 bytes 2023-12-02T22:38:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1975282688 bytes 2023-12-02T22:38:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2002947072 bytes 2023-12-02T22:38:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2018700288 bytes 2023-12-02T22:38:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2058877952 bytes 2023-12-02T22:38:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2123595776 bytes 2023-12-02T22:38:29.419 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:38:29.417+0000 7f4a27c08640 -1 reset not still connected to 0x55cd7e1ed520 2023-12-02T22:38:29.419 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:38:29.417+0000 7f4a27c08640 -1 reset not still connected to 0x55cd8267fa00 2023-12-02T22:38:29.420 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:38:29.418+0000 7f4a27c08640 -1 reset not still connected to 0x55cd8b3c0340 2023-12-02T22:38:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2172191744 bytes 2023-12-02T22:38:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2237172736 bytes 2023-12-02T22:38:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2297901056 bytes 2023-12-02T22:38:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2368401408 bytes 2023-12-02T22:38:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2409956352 bytes 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-12-02T22:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-02T22:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-02T22:38:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_34': 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 140b86879db1 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.140b86879db1 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:30 2023 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:30 2023 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:30 2023 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 72af8c9e-08e6-4b8b-a6d4-3048fd53a1e1 2023-12-02T22:38:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-02T22:38:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:38:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:38:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 55400448 bytes 2023-12-02T22:38:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 77240320 bytes 2023-12-02T22:38:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 163266560 bytes 2023-12-02T22:38:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 214940672 bytes 2023-12-02T22:38:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 258203648 bytes 2023-12-02T22:38:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 273722368 bytes 2023-12-02T22:38:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 331316224 bytes 2023-12-02T22:38:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 365301760 bytes 2023-12-02T22:38:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 390534144 bytes 2023-12-02T22:38:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 423199744 bytes 2023-12-02T22:38:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 434999296 bytes 2023-12-02T22:38:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 447979520 bytes 2023-12-02T22:38:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 495226880 bytes 2023-12-02T22:38:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 578138112 bytes 2023-12-02T22:38:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 642230272 bytes 2023-12-02T22:38:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 678098944 bytes 2023-12-02T22:38:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 783022080 bytes 2023-12-02T22:38:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 823468032 bytes 2023-12-02T22:38:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 865489920 bytes 2023-12-02T22:38:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 917665792 bytes 2023-12-02T22:38:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1039636480 bytes 2023-12-02T22:38:34.263 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:38:34.260+0000 7f2e21bfd640 -1 reset not still connected to 0x557d1b696680 2023-12-02T22:38:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1048658944 bytes 2023-12-02T22:38:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1079910400 bytes 2023-12-02T22:38:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1117651968 bytes 2023-12-02T22:38:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1159076864 bytes 2023-12-02T22:38:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1226814464 bytes 2023-12-02T22:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1294336000 bytes 2023-12-02T22:38:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1302379520 bytes 2023-12-02T22:38:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1330337792 bytes 2023-12-02T22:38:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1359068160 bytes 2023-12-02T22:38:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1395317760 bytes 2023-12-02T22:38:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1438390272 bytes 2023-12-02T22:38:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1502447616 bytes 2023-12-02T22:38:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1519017984 bytes 2023-12-02T22:38:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1578938368 bytes 2023-12-02T22:38:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1666341888 bytes 2023-12-02T22:38:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1747751936 bytes 2023-12-02T22:38:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1804750848 bytes 2023-12-02T22:38:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1812236288 bytes 2023-12-02T22:38:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1842744320 bytes 2023-12-02T22:38:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1867913216 bytes 2023-12-02T22:38:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1884051456 bytes 2023-12-02T22:38:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1925194752 bytes 2023-12-02T22:38:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1999161344 bytes 2023-12-02T22:38:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2060194816 bytes 2023-12-02T22:38:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2159831040 bytes 2023-12-02T22:38:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2187450368 bytes 2023-12-02T22:38:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2261529600 bytes 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2023-12-02T22:38:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-12-02T22:38:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-02T22:38:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-02T22:38:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_35': 2023-12-02T22:38:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1414f3b42726 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1414f3b42726 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:38 2023 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:38 2023 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:38 2023 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4bb9743c-64d8-40e8-b02a-6a91310c2444 2023-12-02T22:38:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-02T22:38:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:38:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:38:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 83225600 bytes 2023-12-02T22:38:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 200728576 bytes 2023-12-02T22:38:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 259958784 bytes 2023-12-02T22:38:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 270078976 bytes 2023-12-02T22:38:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 355207168 bytes 2023-12-02T22:38:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 382024704 bytes 2023-12-02T22:38:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 476828672 bytes 2023-12-02T22:38:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 509475840 bytes 2023-12-02T22:38:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 588565504 bytes 2023-12-02T22:38:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 589171712 bytes 2023-12-02T22:38:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 621862912 bytes 2023-12-02T22:38:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 677921792 bytes 2023-12-02T22:38:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 710093824 bytes 2023-12-02T22:38:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 713434112 bytes 2023-12-02T22:38:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 723855360 bytes 2023-12-02T22:38:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 788993024 bytes 2023-12-02T22:38:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 853724160 bytes 2023-12-02T22:38:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 933507072 bytes 2023-12-02T22:38:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 997741568 bytes 2023-12-02T22:38:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 1037165568 bytes 2023-12-02T22:38:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1093116928 bytes 2023-12-02T22:38:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1118640128 bytes 2023-12-02T22:38:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1134050304 bytes 2023-12-02T22:38:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1220483072 bytes 2023-12-02T22:38:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1221097472 bytes 2023-12-02T22:38:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1266999296 bytes 2023-12-02T22:38:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1283343360 bytes 2023-12-02T22:38:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1379347456 bytes 2023-12-02T22:38:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1442585600 bytes 2023-12-02T22:38:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1507023872 bytes 2023-12-02T22:38:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1554241536 bytes 2023-12-02T22:38:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1614636032 bytes 2023-12-02T22:38:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1657821184 bytes 2023-12-02T22:38:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1687088128 bytes 2023-12-02T22:38:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1728208896 bytes 2023-12-02T22:38:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1798678528 bytes 2023-12-02T22:38:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1849256960 bytes 2023-12-02T22:38:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1946832896 bytes 2023-12-02T22:38:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1994766336 bytes 2023-12-02T22:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 2034605056 bytes 2023-12-02T22:38:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 2051750912 bytes 2023-12-02T22:38:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 2095457280 bytes 2023-12-02T22:38:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2108084224 bytes 2023-12-02T22:38:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2150030336 bytes 2023-12-02T22:38:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2168684544 bytes 2023-12-02T22:38:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2196530176 bytes 2023-12-02T22:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2253494272 bytes 2023-12-02T22:38:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2366366720 bytes 2023-12-02T22:38:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2409196544 bytes 2023-12-02T22:38:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2443880448 bytes 2023-12-02T22:38:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2488553472 bytes 2023-12-02T22:38:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2523938816 bytes 2023-12-02T22:38:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2602622976 bytes 2023-12-02T22:38:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2610222080 bytes 2023-12-02T22:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2662881280 bytes 2023-12-02T22:38:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2700860416 bytes 2023-12-02T22:38:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2722408448 bytes 2023-12-02T22:38:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2744088576 bytes 2023-12-02T22:38:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2757892096 bytes 2023-12-02T22:38:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2779874304 bytes 2023-12-02T22:38:47.302 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T22:38:47.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T22:38:47.302 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-02T22:38:47.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:38:47.301+0000 7fe833eaa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 98770) UID: 1000 2023-12-02T22:38:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2815258624 bytes 2023-12-02T22:38:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2848318464 bytes 2023-12-02T22:38:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-12-02T22:38:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-02T22:38:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:38:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-02T22:38:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_36': 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 141c4895f707 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.141c4895f707 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:47 2023 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:47 2023 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:47 2023 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4b378cd6-dd34-4f37-b8aa-ec8c5a2768b0 2023-12-02T22:38:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-02T22:38:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:38:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:38:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 63790080 bytes 2023-12-02T22:38:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 71987200 bytes 2023-12-02T22:38:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 149089280 bytes 2023-12-02T22:38:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 173579264 bytes 2023-12-02T22:38:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 232014848 bytes 2023-12-02T22:38:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 233302016 bytes 2023-12-02T22:38:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 233913344 bytes 2023-12-02T22:38:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 298203136 bytes 2023-12-02T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 311123968 bytes 2023-12-02T22:38:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 387620864 bytes 2023-12-02T22:38:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 402939904 bytes 2023-12-02T22:38:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 423045120 bytes 2023-12-02T22:38:50.303 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T22:38:50.303 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:38:50.303 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T22:38:50.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:38:50.304 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:38:50.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:38:50.306 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-02T22:38:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 488236032 bytes 2023-12-02T22:38:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 530969600 bytes 2023-12-02T22:38:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 595354624 bytes 2023-12-02T22:38:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 608577536 bytes 2023-12-02T22:38:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 651280384 bytes 2023-12-02T22:38:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 662331392 bytes 2023-12-02T22:38:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 738695168 bytes 2023-12-02T22:38:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 798110720 bytes 2023-12-02T22:38:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 850020352 bytes 2023-12-02T22:38:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 885291008 bytes 2023-12-02T22:38:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 892327936 bytes 2023-12-02T22:38:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 927588352 bytes 2023-12-02T22:38:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 958947328 bytes 2023-12-02T22:38:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 975970304 bytes 2023-12-02T22:38:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1019910144 bytes 2023-12-02T22:38:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1100818432 bytes 2023-12-02T22:38:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1115420672 bytes 2023-12-02T22:38:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1166135296 bytes 2023-12-02T22:38:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1174352896 bytes 2023-12-02T22:38:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1212951552 bytes 2023-12-02T22:38:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1271649280 bytes 2023-12-02T22:38:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1305300992 bytes 2023-12-02T22:38:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1371485184 bytes 2023-12-02T22:38:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1384048640 bytes 2023-12-02T22:38:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1399329792 bytes 2023-12-02T22:38:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1436022784 bytes 2023-12-02T22:38:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1510733824 bytes 2023-12-02T22:38:54.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1596154880 bytes 2023-12-02T22:38:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1676995584 bytes 2023-12-02T22:38:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1685313536 bytes 2023-12-02T22:38:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1708580864 bytes 2023-12-02T22:38:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1740669952 bytes 2023-12-02T22:38:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1791328256 bytes 2023-12-02T22:38:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1811022848 bytes 2023-12-02T22:38:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1837928448 bytes 2023-12-02T22:38:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1890823168 bytes 2023-12-02T22:38:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1946243072 bytes 2023-12-02T22:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1980273664 bytes 2023-12-02T22:38:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2002237440 bytes 2023-12-02T22:38:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2049196032 bytes 2023-12-02T22:38:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2114142208 bytes 2023-12-02T22:38:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2156941312 bytes 2023-12-02T22:38:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2197892096 bytes 2023-12-02T22:38:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2221836288 bytes 2023-12-02T22:38:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2241235968 bytes 2023-12-02T22:38:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2325708800 bytes 2023-12-02T22:38:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2383169536 bytes 2023-12-02T22:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-02T22:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:38:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-12-02T22:38:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:38:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-02T22:38:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:38:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-02T22:38:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_37': 2023-12-02T22:38:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:38:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:38:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 142fd60d705c 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.142fd60d705c 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:56 2023 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:56 2023 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:56 2023 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 6de1f79c-583e-4974-ae84-18b971385b78 2023-12-02T22:38:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:38:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:38:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-02T22:38:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:38:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:38:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:38:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:38:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:38:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:38:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:38:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 96630784 bytes 2023-12-02T22:38:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 137350144 bytes 2023-12-02T22:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 187906048 bytes 2023-12-02T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 211732480 bytes 2023-12-02T22:38:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 297849856 bytes 2023-12-02T22:38:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 319015936 bytes 2023-12-02T22:38:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 364841984 bytes 2023-12-02T22:38:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 395485184 bytes 2023-12-02T22:38:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 439355392 bytes 2023-12-02T22:38:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 473573376 bytes 2023-12-02T22:38:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 492982272 bytes 2023-12-02T22:38:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 572853248 bytes 2023-12-02T22:38:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 604314624 bytes 2023-12-02T22:38:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 651941888 bytes 2023-12-02T22:38:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 652565504 bytes 2023-12-02T22:38:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 667526144 bytes 2023-12-02T22:39:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 682882048 bytes 2023-12-02T22:39:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 698218496 bytes 2023-12-02T22:39:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 721967104 bytes 2023-12-02T22:39:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 791437312 bytes 2023-12-02T22:39:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 827976704 bytes 2023-12-02T22:39:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 838649856 bytes 2023-12-02T22:39:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 924897280 bytes 2023-12-02T22:39:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 954966016 bytes 2023-12-02T22:39:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 993238016 bytes 2023-12-02T22:39:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1018207232 bytes 2023-12-02T22:39:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1038778368 bytes 2023-12-02T22:39:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1054528512 bytes 2023-12-02T22:39:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1120446464 bytes 2023-12-02T22:39:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1193233408 bytes 2023-12-02T22:39:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1268828160 bytes 2023-12-02T22:39:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1294132224 bytes 2023-12-02T22:39:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1381048320 bytes 2023-12-02T22:39:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1388303360 bytes 2023-12-02T22:39:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1455252480 bytes 2023-12-02T22:39:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1471732736 bytes 2023-12-02T22:39:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1514181632 bytes 2023-12-02T22:39:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1540959232 bytes 2023-12-02T22:39:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1605245952 bytes 2023-12-02T22:39:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1639855104 bytes 2023-12-02T22:39:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1697391616 bytes 2023-12-02T22:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1714733056 bytes 2023-12-02T22:39:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1771995136 bytes 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:39:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:39:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:39:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-02T22:39:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:39:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_38': 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 14398303f424 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.14398303f424 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:39:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:03 2023 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:03 2023 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:03 2023 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dedd0af-fa04-42db-9022-5c3e623cc7a9 2023-12-02T22:39:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:39:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:39:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-02T22:39:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:39:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:39:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:39:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:39:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:39:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 616448 bytes 2023-12-02T22:39:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 25479168 bytes 2023-12-02T22:39:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 49729536 bytes 2023-12-02T22:39:05.194 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T22:39:05.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T22:39:05.194 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:39:05.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:39:05.195+0000 7fd723137640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 98618) UID: 1000 2023-12-02T22:39:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 91375616 bytes 2023-12-02T22:39:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 202499072 bytes 2023-12-02T22:39:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 233566208 bytes 2023-12-02T22:39:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 296393728 bytes 2023-12-02T22:39:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 374327296 bytes 2023-12-02T22:39:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 405895168 bytes 2023-12-02T22:39:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 442892288 bytes 2023-12-02T22:39:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 579459072 bytes 2023-12-02T22:39:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 589383680 bytes 2023-12-02T22:39:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 658566144 bytes 2023-12-02T22:39:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 738878464 bytes 2023-12-02T22:39:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 748627968 bytes 2023-12-02T22:39:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 755193856 bytes 2023-12-02T22:39:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 844036096 bytes 2023-12-02T22:39:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 857588736 bytes 2023-12-02T22:39:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 900719616 bytes 2023-12-02T22:39:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 942960640 bytes 2023-12-02T22:39:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 981537792 bytes 2023-12-02T22:39:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1050658816 bytes 2023-12-02T22:39:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1081320448 bytes 2023-12-02T22:39:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1140775936 bytes 2023-12-02T22:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1202928640 bytes 2023-12-02T22:39:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1203556352 bytes 2023-12-02T22:39:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1251942400 bytes 2023-12-02T22:39:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1252557824 bytes 2023-12-02T22:39:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1294727168 bytes 2023-12-02T22:39:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1323757568 bytes 2023-12-02T22:39:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1390914560 bytes 2023-12-02T22:39:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1407814656 bytes 2023-12-02T22:39:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1424976896 bytes 2023-12-02T22:39:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1435374592 bytes 2023-12-02T22:39:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1442155520 bytes 2023-12-02T22:39:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1488077824 bytes 2023-12-02T22:39:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1523247104 bytes 2023-12-02T22:39:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1534403584 bytes 2023-12-02T22:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1575788544 bytes 2023-12-02T22:39:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1607263232 bytes 2023-12-02T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1653030912 bytes 2023-12-02T22:39:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1723393024 bytes 2023-12-02T22:39:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1794245632 bytes 2023-12-02T22:39:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1853451264 bytes 2023-12-02T22:39:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1952221184 bytes 2023-12-02T22:39:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1987419136 bytes 2023-12-02T22:39:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2034741248 bytes 2023-12-02T22:39:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2091362304 bytes 2023-12-02T22:39:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2149397504 bytes 2023-12-02T22:39:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2168138752 bytes 2023-12-02T22:39:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2207735808 bytes 2023-12-02T22:39:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2284275712 bytes 2023-12-02T22:39:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2324501504 bytes 2023-12-02T22:39:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2412228608 bytes 2023-12-02T22:39:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2437913600 bytes 2023-12-02T22:39:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2505042944 bytes 2023-12-02T22:39:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2546702336 bytes 2023-12-02T22:39:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2632146944 bytes 2023-12-02T22:39:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2676485120 bytes 2023-12-02T22:39:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2696818688 bytes 2023-12-02T22:39:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2727756800 bytes 2023-12-02T22:39:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2770589696 bytes 2023-12-02T22:39:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2838183936 bytes 2023-12-02T22:39:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2870103040 bytes 2023-12-02T22:39:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2922297344 bytes 2023-12-02T22:39:14.195 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T22:39:14.195 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:39:14.195 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T22:39:14.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T22:39:14.196 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T22:39:14.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T22:39:14.198 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-02T22:39:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 3027158016 bytes 2023-12-02T22:39:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3065923584 bytes 2023-12-02T22:39:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3106523136 bytes 2023-12-02T22:39:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3246558208 bytes 2023-12-02T22:39:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3282306048 bytes 2023-12-02T22:39:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3370991616 bytes 2023-12-02T22:39:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3371611136 bytes 2023-12-02T22:39:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3407309824 bytes 2023-12-02T22:39:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3445023744 bytes 2023-12-02T22:39:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3487338496 bytes 2023-12-02T22:39:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3504028672 bytes 2023-12-02T22:39:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3530443776 bytes 2023-12-02T22:39:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3577166848 bytes 2023-12-02T22:39:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3588657152 bytes 2023-12-02T22:39:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3605640192 bytes 2023-12-02T22:39:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3682525184 bytes 2023-12-02T22:39:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3715056640 bytes 2023-12-02T22:39:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3740476416 bytes 2023-12-02T22:39:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3755646976 bytes 2023-12-02T22:39:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:39:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:39:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:39:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:39:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-02T22:39:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-12-02T22:39:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:39:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:39:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-02T22:39:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:39:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-02T22:39:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:39:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_39': 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 144868a9a41c 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.144868a9a41c 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:16 2023 2023-12-02T22:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:16 2023 2023-12-02T22:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:16 2023 2023-12-02T22:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e5ea4907-8178-4427-a782-ee8006b75e82 2023-12-02T22:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-02T22:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:39:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:39:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2023-12-02T22:39:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:39:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:39:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 40210432 bytes 2023-12-02T22:39:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 108296192 bytes 2023-12-02T22:39:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 169023488 bytes 2023-12-02T22:39:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 227326976 bytes 2023-12-02T22:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 301605888 bytes 2023-12-02T22:39:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 356910080 bytes 2023-12-02T22:39:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 400509952 bytes 2023-12-02T22:39:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 437891072 bytes 2023-12-02T22:39:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 570788864 bytes 2023-12-02T22:39:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 619478016 bytes 2023-12-02T22:39:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 655740928 bytes 2023-12-02T22:39:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 734444544 bytes 2023-12-02T22:39:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 786433024 bytes 2023-12-02T22:39:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 881471488 bytes 2023-12-02T22:39:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 898684928 bytes 2023-12-02T22:39:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 904628224 bytes 2023-12-02T22:39:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 905248768 bytes 2023-12-02T22:39:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 972680192 bytes 2023-12-02T22:39:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 1039700992 bytes 2023-12-02T22:39:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 1102616576 bytes 2023-12-02T22:39:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1182731264 bytes 2023-12-02T22:39:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1210720256 bytes 2023-12-02T22:39:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1292723200 bytes 2023-12-02T22:39:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1378942976 bytes 2023-12-02T22:39:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1417706496 bytes 2023-12-02T22:39:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1469478912 bytes 2023-12-02T22:39:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1552309248 bytes 2023-12-02T22:39:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1614539776 bytes 2023-12-02T22:39:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1667604480 bytes 2023-12-02T22:39:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1688095744 bytes 2023-12-02T22:39:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1713461248 bytes 2023-12-02T22:39:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1787727872 bytes 2023-12-02T22:39:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1850942464 bytes 2023-12-02T22:39:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1931043840 bytes 2023-12-02T22:39:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 2010890240 bytes 2023-12-02T22:39:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 2053628928 bytes 2023-12-02T22:39:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 2120118272 bytes 2023-12-02T22:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 2189448192 bytes 2023-12-02T22:39:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 2221832192 bytes 2023-12-02T22:39:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 2258166784 bytes 2023-12-02T22:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 2288322560 bytes 2023-12-02T22:39:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 2300989440 bytes 2023-12-02T22:39:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2313067520 bytes 2023-12-02T22:39:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2350759936 bytes 2023-12-02T22:39:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2422237184 bytes 2023-12-02T22:39:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2462131200 bytes 2023-12-02T22:39:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2473324544 bytes 2023-12-02T22:39:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2565605376 bytes 2023-12-02T22:39:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2654548992 bytes 2023-12-02T22:39:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2671805440 bytes 2023-12-02T22:39:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2690764800 bytes 2023-12-02T22:39:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2734383104 bytes 2023-12-02T22:39:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2753620992 bytes 2023-12-02T22:39:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2794163200 bytes 2023-12-02T22:39:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2841912320 bytes 2023-12-02T22:39:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2894305280 bytes 2023-12-02T22:39:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2942619648 bytes 2023-12-02T22:39:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2999618560 bytes 2023-12-02T22:39:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 3074761728 bytes 2023-12-02T22:39:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 3100237824 bytes 2023-12-02T22:39:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 3212168192 bytes 2023-12-02T22:39:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 3288107008 bytes 2023-12-02T22:39:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 3333165056 bytes 2023-12-02T22:39:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 3356849152 bytes 2023-12-02T22:39:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 3372637184 bytes 2023-12-02T22:39:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 3394164736 bytes 2023-12-02T22:39:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3399950336 bytes 2023-12-02T22:39:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3427924992 bytes 2023-12-02T22:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3467515904 bytes 2023-12-02T22:39:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3563265024 bytes 2023-12-02T22:39:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3625611264 bytes 2023-12-02T22:39:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3657704448 bytes 2023-12-02T22:39:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3718602752 bytes 2023-12-02T22:39:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3759017984 bytes 2023-12-02T22:39:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3774992384 bytes 2023-12-02T22:39:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3794780160 bytes 2023-12-02T22:39:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3806667776 bytes 2023-12-02T22:39:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3874831360 bytes 2023-12-02T22:39:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3916106752 bytes 2023-12-02T22:39:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3967118336 bytes 2023-12-02T22:39:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 4009158656 bytes 2023-12-02T22:39:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 4026754048 bytes 2023-12-02T22:39:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 4051398656 bytes 2023-12-02T22:39:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 4118915072 bytes 2023-12-02T22:39:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 4119517184 bytes 2023-12-02T22:39:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 4280151040 bytes 2023-12-02T22:39:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 4356610048 bytes 2023-12-02T22:39:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:39:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:39:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:39:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-12-02T22:39:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-02T22:39:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:39:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_40': 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 145078e8a61f 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.145078e8a61f 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:30 2023 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:30 2023 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:30 2023 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7f8c70ee-6301-46d2-8c34-89c793426067 2023-12-02T22:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-02T22:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:39:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:39:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:39:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:39:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:39:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 73341952 bytes 2023-12-02T22:39:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 101037056 bytes 2023-12-02T22:39:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 173762560 bytes 2023-12-02T22:39:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 174910464 bytes 2023-12-02T22:39:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 218351616 bytes 2023-12-02T22:39:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 311536640 bytes 2023-12-02T22:39:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 337829888 bytes 2023-12-02T22:39:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 380869632 bytes 2023-12-02T22:39:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 435091456 bytes 2023-12-02T22:39:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 456404992 bytes 2023-12-02T22:39:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 479381504 bytes 2023-12-02T22:39:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 555463680 bytes 2023-12-02T22:39:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 621240320 bytes 2023-12-02T22:39:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 727264256 bytes 2023-12-02T22:39:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 762926080 bytes 2023-12-02T22:39:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 829462528 bytes 2023-12-02T22:39:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 893052928 bytes 2023-12-02T22:39:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 918172672 bytes 2023-12-02T22:39:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 974450688 bytes 2023-12-02T22:39:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 996648960 bytes 2023-12-02T22:39:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1062602752 bytes 2023-12-02T22:39:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1094345728 bytes 2023-12-02T22:39:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1113350144 bytes 2023-12-02T22:39:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1154490368 bytes 2023-12-02T22:39:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1172622336 bytes 2023-12-02T22:39:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1200380928 bytes 2023-12-02T22:39:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1201444864 bytes 2023-12-02T22:39:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1271352320 bytes 2023-12-02T22:39:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1297077248 bytes 2023-12-02T22:39:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1405540352 bytes 2023-12-02T22:39:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1454189568 bytes 2023-12-02T22:39:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1481460736 bytes 2023-12-02T22:39:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1556128768 bytes 2023-12-02T22:39:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1596813312 bytes 2023-12-02T22:39:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1613371392 bytes 2023-12-02T22:39:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1663730688 bytes 2023-12-02T22:39:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1677815808 bytes 2023-12-02T22:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1684301824 bytes 2023-12-02T22:39:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1748367360 bytes 2023-12-02T22:39:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1751903232 bytes 2023-12-02T22:39:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1791092736 bytes 2023-12-02T22:39:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1815668736 bytes 2023-12-02T22:39:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1848132608 bytes 2023-12-02T22:39:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1858537472 bytes 2023-12-02T22:39:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1905077248 bytes 2023-12-02T22:39:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2024876032 bytes 2023-12-02T22:39:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2088091648 bytes 2023-12-02T22:39:37.853 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:39:37.850+0000 7f315ad82640 -1 reset not still connected to 0x564df1a4d930 2023-12-02T22:39:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2107518976 bytes 2023-12-02T22:39:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2156605440 bytes 2023-12-02T22:39:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2210859008 bytes 2023-12-02T22:39:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2238217216 bytes 2023-12-02T22:39:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2267530240 bytes 2023-12-02T22:39:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2281840640 bytes 2023-12-02T22:39:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2317813760 bytes 2023-12-02T22:39:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2359932928 bytes 2023-12-02T22:39:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2385569792 bytes 2023-12-02T22:39:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2422254592 bytes 2023-12-02T22:39:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:39:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:39:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:39:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-02T22:39:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-02T22:39:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:39:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-02T22:39:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_41': 2023-12-02T22:39:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:39:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:39:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:39:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1459c669df87 2023-12-02T22:39:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1459c669df87 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:39 2023 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:39 2023 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:39 2023 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a49a8b18-32ec-4784-9e10-57e3c1d4cf26 2023-12-02T22:39:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:39:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:39:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-02T22:39:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:39:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:39:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:39:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:39:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:39:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 24456192 bytes 2023-12-02T22:39:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 75086848 bytes 2023-12-02T22:39:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 107926528 bytes 2023-12-02T22:39:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 135346176 bytes 2023-12-02T22:39:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 157806592 bytes 2023-12-02T22:39:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 214358016 bytes 2023-12-02T22:39:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 260482048 bytes 2023-12-02T22:39:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 321013760 bytes 2023-12-02T22:39:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 321635328 bytes 2023-12-02T22:39:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 361757696 bytes 2023-12-02T22:39:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 380537856 bytes 2023-12-02T22:39:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 446687232 bytes 2023-12-02T22:39:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 484354048 bytes 2023-12-02T22:39:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 510565376 bytes 2023-12-02T22:39:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 579944448 bytes 2023-12-02T22:39:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 596158464 bytes 2023-12-02T22:39:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 640960512 bytes 2023-12-02T22:39:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 668124160 bytes 2023-12-02T22:39:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 738641920 bytes 2023-12-02T22:39:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 783468544 bytes 2023-12-02T22:39:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 838162432 bytes 2023-12-02T22:39:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 883837952 bytes 2023-12-02T22:39:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 918600704 bytes 2023-12-02T22:39:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 975790080 bytes 2023-12-02T22:39:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1014809600 bytes 2023-12-02T22:39:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1015429120 bytes 2023-12-02T22:39:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1035572224 bytes 2023-12-02T22:39:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1061841920 bytes 2023-12-02T22:39:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1094349824 bytes 2023-12-02T22:39:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1094967296 bytes 2023-12-02T22:39:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1177907200 bytes 2023-12-02T22:39:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1233996800 bytes 2023-12-02T22:39:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1234606080 bytes 2023-12-02T22:39:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1282470912 bytes 2023-12-02T22:39:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1302773760 bytes 2023-12-02T22:39:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1347937280 bytes 2023-12-02T22:39:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1376956416 bytes 2023-12-02T22:39:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1423626240 bytes 2023-12-02T22:39:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1451372544 bytes 2023-12-02T22:39:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1467334656 bytes 2023-12-02T22:39:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1486071808 bytes 2023-12-02T22:39:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1495494656 bytes 2023-12-02T22:39:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1541402624 bytes 2023-12-02T22:39:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1564562432 bytes 2023-12-02T22:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1592800256 bytes 2023-12-02T22:39:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1593409536 bytes 2023-12-02T22:39:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1630830592 bytes 2023-12-02T22:39:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1631951872 bytes 2023-12-02T22:39:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1661448192 bytes 2023-12-02T22:39:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1707206656 bytes 2023-12-02T22:39:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1728794624 bytes 2023-12-02T22:39:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1786640384 bytes 2023-12-02T22:39:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1827123200 bytes 2023-12-02T22:39:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 1866631168 bytes 2023-12-02T22:39:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 1923518464 bytes 2023-12-02T22:39:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 1935509504 bytes 2023-12-02T22:39:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 1936135168 bytes 2023-12-02T22:39:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 1970189312 bytes 2023-12-02T22:39:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2001698816 bytes 2023-12-02T22:39:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2007390208 bytes 2023-12-02T22:39:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2024265728 bytes 2023-12-02T22:39:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2068812800 bytes 2023-12-02T22:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2101625856 bytes 2023-12-02T22:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2136630272 bytes 2023-12-02T22:39:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2166460416 bytes 2023-12-02T22:39:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2185318400 bytes 2023-12-02T22:39:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2260768768 bytes 2023-12-02T22:39:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2305350656 bytes 2023-12-02T22:39:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2327812096 bytes 2023-12-02T22:39:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2374472704 bytes 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:39:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:39:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:39:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:39:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-12-02T22:39:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:39:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:39:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-02T22:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-02T22:39:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:39:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-02T22:39:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_42': 2023-12-02T22:39:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:39:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:39:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:39:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1461f942e0cb 2023-12-02T22:39:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1461f942e0cb 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:48 2023 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:48 2023 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:48 2023 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7d8c8ff7-9e39-4fd4-a378-75b3505c1c8c 2023-12-02T22:39:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:39:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:39:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-02T22:39:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:39:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:39:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:39:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:39:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:39:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 88424448 bytes 2023-12-02T22:39:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 106349568 bytes 2023-12-02T22:39:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 158279680 bytes 2023-12-02T22:39:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 212613120 bytes 2023-12-02T22:39:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 254774272 bytes 2023-12-02T22:39:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 269787136 bytes 2023-12-02T22:39:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 316316672 bytes 2023-12-02T22:39:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 332359680 bytes 2023-12-02T22:39:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 352616448 bytes 2023-12-02T22:39:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 435931136 bytes 2023-12-02T22:39:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 461793280 bytes 2023-12-02T22:39:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 527879168 bytes 2023-12-02T22:39:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 605379584 bytes 2023-12-02T22:39:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 674104320 bytes 2023-12-02T22:39:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 712376320 bytes 2023-12-02T22:39:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 738233344 bytes 2023-12-02T22:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 778032128 bytes 2023-12-02T22:39:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 782417920 bytes 2023-12-02T22:39:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 829808640 bytes 2023-12-02T22:39:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 879181824 bytes 2023-12-02T22:39:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 933107712 bytes 2023-12-02T22:39:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1000096768 bytes 2023-12-02T22:39:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1072519168 bytes 2023-12-02T22:39:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1147063296 bytes 2023-12-02T22:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1182052352 bytes 2023-12-02T22:39:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1218096128 bytes 2023-12-02T22:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1261458432 bytes 2023-12-02T22:39:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1304978432 bytes 2023-12-02T22:39:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1372419072 bytes 2023-12-02T22:39:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1393551360 bytes 2023-12-02T22:39:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1394181120 bytes 2023-12-02T22:39:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1403695104 bytes 2023-12-02T22:39:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1431751680 bytes 2023-12-02T22:39:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1452959744 bytes 2023-12-02T22:39:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1497338880 bytes 2023-12-02T22:39:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1538169856 bytes 2023-12-02T22:39:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1582828544 bytes 2023-12-02T22:39:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1601746944 bytes 2023-12-02T22:39:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1677237248 bytes 2023-12-02T22:39:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1792617472 bytes 2023-12-02T22:39:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1826742272 bytes 2023-12-02T22:39:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1865381888 bytes 2023-12-02T22:39:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1882956800 bytes 2023-12-02T22:39:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2008318976 bytes 2023-12-02T22:39:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2015675392 bytes 2023-12-02T22:39:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2025180160 bytes 2023-12-02T22:39:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2061688832 bytes 2023-12-02T22:39:56.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2087384064 bytes 2023-12-02T22:39:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2129128448 bytes 2023-12-02T22:39:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2152022016 bytes 2023-12-02T22:39:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2170597376 bytes 2023-12-02T22:39:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2215211008 bytes 2023-12-02T22:39:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2247717888 bytes 2023-12-02T22:39:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2272184320 bytes 2023-12-02T22:39:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2310098944 bytes 2023-12-02T22:39:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2379833344 bytes 2023-12-02T22:39:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2383169536 bytes 2023-12-02T22:39:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2404569088 bytes 2023-12-02T22:39:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2543334400 bytes 2023-12-02T22:39:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2573812736 bytes 2023-12-02T22:39:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2575011840 bytes 2023-12-02T22:39:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2636001280 bytes 2023-12-02T22:39:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2719960064 bytes 2023-12-02T22:39:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2735448064 bytes 2023-12-02T22:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2764507136 bytes 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:39:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:39:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:39:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-12-02T22:39:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:39:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-02T22:39:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:39:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-02T22:39:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_43': 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 146af6985997 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.146af6985997 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:58 2023 2023-12-02T22:39:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:58 2023 2023-12-02T22:39:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:58 2023 2023-12-02T22:39:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:39:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:39:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: dabad757-1fd3-41ef-b3ef-13b1187bb4f2 2023-12-02T22:39:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:39:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:39:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-02T22:39:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:39:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:39:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:39:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:39:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:39:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:39:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:39:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 35780608 bytes 2023-12-02T22:40:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 61729792 bytes 2023-12-02T22:40:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 115235840 bytes 2023-12-02T22:40:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 177195008 bytes 2023-12-02T22:40:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 278643712 bytes 2023-12-02T22:40:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 303340544 bytes 2023-12-02T22:40:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 352952320 bytes 2023-12-02T22:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 405325824 bytes 2023-12-02T22:40:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 491577344 bytes 2023-12-02T22:40:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 549657600 bytes 2023-12-02T22:40:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 577977344 bytes 2023-12-02T22:40:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 598167552 bytes 2023-12-02T22:40:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 623577088 bytes 2023-12-02T22:40:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 685158400 bytes 2023-12-02T22:40:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 811132928 bytes 2023-12-02T22:40:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 859072512 bytes 2023-12-02T22:40:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 859668480 bytes 2023-12-02T22:40:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 907995136 bytes 2023-12-02T22:40:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 918273024 bytes 2023-12-02T22:40:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 971815936 bytes 2023-12-02T22:40:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1072690176 bytes 2023-12-02T22:40:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1081564160 bytes 2023-12-02T22:40:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1110275072 bytes 2023-12-02T22:40:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1116240896 bytes 2023-12-02T22:40:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1152885760 bytes 2023-12-02T22:40:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1188288512 bytes 2023-12-02T22:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1202684928 bytes 2023-12-02T22:40:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1274994688 bytes 2023-12-02T22:40:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1317202944 bytes 2023-12-02T22:40:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1360330752 bytes 2023-12-02T22:40:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1401733120 bytes 2023-12-02T22:40:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1420285952 bytes 2023-12-02T22:40:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1420936192 bytes 2023-12-02T22:40:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1429328896 bytes 2023-12-02T22:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1507768320 bytes 2023-12-02T22:40:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1571750912 bytes 2023-12-02T22:40:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1584202752 bytes 2023-12-02T22:40:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1644614656 bytes 2023-12-02T22:40:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1665516544 bytes 2023-12-02T22:40:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1689331712 bytes 2023-12-02T22:40:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1719965696 bytes 2023-12-02T22:40:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1748291584 bytes 2023-12-02T22:40:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1833249792 bytes 2023-12-02T22:40:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1851214848 bytes 2023-12-02T22:40:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1882961920 bytes 2023-12-02T22:40:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1907198976 bytes 2023-12-02T22:40:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1966192640 bytes 2023-12-02T22:40:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1977293824 bytes 2023-12-02T22:40:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2008394752 bytes 2023-12-02T22:40:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2048172032 bytes 2023-12-02T22:40:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-12-02T22:40:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-02T22:40:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-02T22:40:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_44': 2023-12-02T22:40:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 14751b06534 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.14751b06534 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:06 2023 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:06 2023 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:06 2023 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a442254e-d4a3-4fcf-9c21-028b8a99b2f0 2023-12-02T22:40:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-02T22:40:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:40:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:40:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 20445184 bytes 2023-12-02T22:40:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 44507136 bytes 2023-12-02T22:40:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 75739136 bytes 2023-12-02T22:40:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 101440512 bytes 2023-12-02T22:40:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 137604096 bytes 2023-12-02T22:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 185095168 bytes 2023-12-02T22:40:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 240307200 bytes 2023-12-02T22:40:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 262349824 bytes 2023-12-02T22:40:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 275195904 bytes 2023-12-02T22:40:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 295865344 bytes 2023-12-02T22:40:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 317146112 bytes 2023-12-02T22:40:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 319634432 bytes 2023-12-02T22:40:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 397489152 bytes 2023-12-02T22:40:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 420855808 bytes 2023-12-02T22:40:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 432357376 bytes 2023-12-02T22:40:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 490191872 bytes 2023-12-02T22:40:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 509690880 bytes 2023-12-02T22:40:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 555656192 bytes 2023-12-02T22:40:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 603982848 bytes 2023-12-02T22:40:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 604599296 bytes 2023-12-02T22:40:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 642255872 bytes 2023-12-02T22:40:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 731748352 bytes 2023-12-02T22:40:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 774195200 bytes 2023-12-02T22:40:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 828846080 bytes 2023-12-02T22:40:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 883484672 bytes 2023-12-02T22:40:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 919195648 bytes 2023-12-02T22:40:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 938089472 bytes 2023-12-02T22:40:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 967927808 bytes 2023-12-02T22:40:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 997892096 bytes 2023-12-02T22:40:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1032121344 bytes 2023-12-02T22:40:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1074761728 bytes 2023-12-02T22:40:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1115738112 bytes 2023-12-02T22:40:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1128941568 bytes 2023-12-02T22:40:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1174969344 bytes 2023-12-02T22:40:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1175586816 bytes 2023-12-02T22:40:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1197282304 bytes 2023-12-02T22:40:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1252082688 bytes 2023-12-02T22:40:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1258044416 bytes 2023-12-02T22:40:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1345154048 bytes 2023-12-02T22:40:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1345770496 bytes 2023-12-02T22:40:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1372143616 bytes 2023-12-02T22:40:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1441885184 bytes 2023-12-02T22:40:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1456911360 bytes 2023-12-02T22:40:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1507423232 bytes 2023-12-02T22:40:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1509981184 bytes 2023-12-02T22:40:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1584749568 bytes 2023-12-02T22:40:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1607350272 bytes 2023-12-02T22:40:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1727283200 bytes 2023-12-02T22:40:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1800075264 bytes 2023-12-02T22:40:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1866462208 bytes 2023-12-02T22:40:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1948477440 bytes 2023-12-02T22:40:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1964212224 bytes 2023-12-02T22:40:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1983362048 bytes 2023-12-02T22:40:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2006679552 bytes 2023-12-02T22:40:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2045463552 bytes 2023-12-02T22:40:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2094090240 bytes 2023-12-02T22:40:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2121445376 bytes 2023-12-02T22:40:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2161575936 bytes 2023-12-02T22:40:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2230107136 bytes 2023-12-02T22:40:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2293509120 bytes 2023-12-02T22:40:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2325769216 bytes 2023-12-02T22:40:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2332235776 bytes 2023-12-02T22:40:16.199 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7742637691613309) > weight (0.25) 2023-12-02T22:40:16.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9542685181936406) > weight (0.25) 2023-12-02T22:40:16.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7324407544714194) > weight (0.25) 2023-12-02T22:40:16.200 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T22:40:16.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T22:40:16.201 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T22:40:16.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:40:16.199+0000 7f1d84ae3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 94685) UID: 1000 2023-12-02T22:40:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2441701376 bytes 2023-12-02T22:40:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2496038912 bytes 2023-12-02T22:40:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2554941440 bytes 2023-12-02T22:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2611931136 bytes 2023-12-02T22:40:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-02T22:40:17.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:40:17.396+0000 7f573d240640 -1 rbd::mirror::ImageReplayer: 0x561a1f5fcf00 [3/645a0ff2-7b25-4678-9148-2db63c48f624] start: already running 2023-12-02T22:40:17.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:40:17.396+0000 7f573d240640 -1 rbd::mirror::ImageReplayer: 0x561a1f5fdb80 [3/4bb9743c-64d8-40e8-b02a-6a91310c2444] start: already running 2023-12-02T22:40:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_45': 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 147f95aba389 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.147f95aba389 2023-12-02T22:40:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:16 2023 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:16 2023 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:16 2023 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 09f87dc1-458a-4e3a-a6c6-7afee7b3129e 2023-12-02T22:40:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-02T22:40:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2023-12-02T22:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 23657472 bytes 2023-12-02T22:40:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 85716992 bytes 2023-12-02T22:40:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 170720256 bytes 2023-12-02T22:40:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 214641664 bytes 2023-12-02T22:40:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 235678720 bytes 2023-12-02T22:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 321494016 bytes 2023-12-02T22:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 371809280 bytes 2023-12-02T22:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 453529600 bytes 2023-12-02T22:40:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 522319872 bytes 2023-12-02T22:40:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 528093184 bytes 2023-12-02T22:40:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 563809280 bytes 2023-12-02T22:40:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 577776640 bytes 2023-12-02T22:40:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 614094848 bytes 2023-12-02T22:40:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 632555520 bytes 2023-12-02T22:40:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 642153472 bytes 2023-12-02T22:40:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 736980992 bytes 2023-12-02T22:40:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 763719680 bytes 2023-12-02T22:40:20.201 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T22:40:20.202 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:40:20.202 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T22:40:20.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T22:40:20.202 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T22:40:20.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T22:40:20.206 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-02T22:40:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 848930816 bytes 2023-12-02T22:40:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 902027264 bytes 2023-12-02T22:40:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 923193344 bytes 2023-12-02T22:40:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 988564480 bytes 2023-12-02T22:40:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1025728512 bytes 2023-12-02T22:40:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1039919104 bytes 2023-12-02T22:40:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1084264448 bytes 2023-12-02T22:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1134931968 bytes 2023-12-02T22:40:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1156373504 bytes 2023-12-02T22:40:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1214910464 bytes 2023-12-02T22:40:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1278909440 bytes 2023-12-02T22:40:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1320743936 bytes 2023-12-02T22:40:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1335649280 bytes 2023-12-02T22:40:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1378262016 bytes 2023-12-02T22:40:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1481268224 bytes 2023-12-02T22:40:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1532432384 bytes 2023-12-02T22:40:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1599575040 bytes 2023-12-02T22:40:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-12-02T22:40:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-02T22:40:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:40:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-02T22:40:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-02T22:40:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_46': 2023-12-02T22:40:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 148d859a8501 2023-12-02T22:40:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.148d859a8501 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:22 2023 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:22 2023 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:22 2023 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c1d2a65d-c465-4d9a-8cce-99526d99432c 2023-12-02T22:40:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-02T22:40:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:40:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2023-12-02T22:40:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 91110400 bytes 2023-12-02T22:40:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 179399680 bytes 2023-12-02T22:40:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 200555520 bytes 2023-12-02T22:40:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 274711552 bytes 2023-12-02T22:40:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 304225280 bytes 2023-12-02T22:40:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 354800640 bytes 2023-12-02T22:40:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 402702336 bytes 2023-12-02T22:40:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 404626432 bytes 2023-12-02T22:40:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 434664448 bytes 2023-12-02T22:40:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 447128576 bytes 2023-12-02T22:40:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 459044864 bytes 2023-12-02T22:40:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 479874048 bytes 2023-12-02T22:40:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 517961728 bytes 2023-12-02T22:40:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 570761216 bytes 2023-12-02T22:40:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 602910720 bytes 2023-12-02T22:40:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 681820160 bytes 2023-12-02T22:40:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 708622336 bytes 2023-12-02T22:40:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 798800896 bytes 2023-12-02T22:40:26.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7552700920328445) > weight (0.25) 2023-12-02T22:40:26.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6772750654912821) > weight (0.25) 2023-12-02T22:40:26.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7195230071971943) > weight (0.25) 2023-12-02T22:40:26.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9288508590842834) > weight (0.25) 2023-12-02T22:40:26.308 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-02T22:40:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 851482624 bytes 2023-12-02T22:40:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 892495872 bytes 2023-12-02T22:40:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 910765056 bytes 2023-12-02T22:40:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 921877504 bytes 2023-12-02T22:40:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 936862720 bytes 2023-12-02T22:40:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 946570240 bytes 2023-12-02T22:40:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1064474624 bytes 2023-12-02T22:40:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1130084352 bytes 2023-12-02T22:40:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1137353728 bytes 2023-12-02T22:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1169045504 bytes 2023-12-02T22:40:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1206302720 bytes 2023-12-02T22:40:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1299502080 bytes 2023-12-02T22:40:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1390898176 bytes 2023-12-02T22:40:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1450286080 bytes 2023-12-02T22:40:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1476488192 bytes 2023-12-02T22:40:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1568375808 bytes 2023-12-02T22:40:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1570869248 bytes 2023-12-02T22:40:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1586186240 bytes 2023-12-02T22:40:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1586797568 bytes 2023-12-02T22:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1597916160 bytes 2023-12-02T22:40:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1615558656 bytes 2023-12-02T22:40:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1654117376 bytes 2023-12-02T22:40:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1693077504 bytes 2023-12-02T22:40:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1741188096 bytes 2023-12-02T22:40:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1830900736 bytes 2023-12-02T22:40:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1863887872 bytes 2023-12-02T22:40:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1938169856 bytes 2023-12-02T22:40:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1977378816 bytes 2023-12-02T22:40:30.129 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:40:30.126+0000 7f2e21bfd640 -1 reset not still connected to 0x557cdd9f7a00 2023-12-02T22:40:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2043582464 bytes 2023-12-02T22:40:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2085800960 bytes 2023-12-02T22:40:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2116953088 bytes 2023-12-02T22:40:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2174434304 bytes 2023-12-02T22:40:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2229746688 bytes 2023-12-02T22:40:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2240294912 bytes 2023-12-02T22:40:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2277125120 bytes 2023-12-02T22:40:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2322810880 bytes 2023-12-02T22:40:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2365326336 bytes 2023-12-02T22:40:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2416622592 bytes 2023-12-02T22:40:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2440691712 bytes 2023-12-02T22:40:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2442760192 bytes 2023-12-02T22:40:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2495475712 bytes 2023-12-02T22:40:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2541091840 bytes 2023-12-02T22:40:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2570067968 bytes 2023-12-02T22:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2623551488 bytes 2023-12-02T22:40:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2679791616 bytes 2023-12-02T22:40:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2686968832 bytes 2023-12-02T22:40:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2768887808 bytes 2023-12-02T22:40:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2818793472 bytes 2023-12-02T22:40:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2885820416 bytes 2023-12-02T22:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2908926976 bytes 2023-12-02T22:40:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2984863744 bytes 2023-12-02T22:40:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3008044032 bytes 2023-12-02T22:40:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3032153088 bytes 2023-12-02T22:40:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3083315200 bytes 2023-12-02T22:40:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3102233600 bytes 2023-12-02T22:40:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3161347072 bytes 2023-12-02T22:40:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3190296576 bytes 2023-12-02T22:40:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3222066176 bytes 2023-12-02T22:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3262425088 bytes 2023-12-02T22:40:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3275307008 bytes 2023-12-02T22:40:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3312412672 bytes 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:40:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-12-02T22:40:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-02T22:40:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:40:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-02T22:40:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_47': 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1496b9aa5586 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1496b9aa5586 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:34 2023 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:34 2023 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:34 2023 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 9431194f-1321-4cad-a347-45f6490d98bd 2023-12-02T22:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-02T22:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2023-12-02T22:40:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 22903808 bytes 2023-12-02T22:40:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 40695808 bytes 2023-12-02T22:40:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 43459584 bytes 2023-12-02T22:40:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 50852864 bytes 2023-12-02T22:40:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 67890176 bytes 2023-12-02T22:40:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 76677120 bytes 2023-12-02T22:40:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 183184384 bytes 2023-12-02T22:40:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 214019072 bytes 2023-12-02T22:40:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 243674112 bytes 2023-12-02T22:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 256964608 bytes 2023-12-02T22:40:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 304958464 bytes 2023-12-02T22:40:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 310039552 bytes 2023-12-02T22:40:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 367644672 bytes 2023-12-02T22:40:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 403625984 bytes 2023-12-02T22:40:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 445589504 bytes 2023-12-02T22:40:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 467827712 bytes 2023-12-02T22:40:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 505148416 bytes 2023-12-02T22:40:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 513934336 bytes 2023-12-02T22:40:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 544662528 bytes 2023-12-02T22:40:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 596891648 bytes 2023-12-02T22:40:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 666405888 bytes 2023-12-02T22:40:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 687549440 bytes 2023-12-02T22:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 730767360 bytes 2023-12-02T22:40:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 784609280 bytes 2023-12-02T22:40:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 860257280 bytes 2023-12-02T22:40:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 867894272 bytes 2023-12-02T22:40:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 888584192 bytes 2023-12-02T22:40:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 898870272 bytes 2023-12-02T22:40:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 922774528 bytes 2023-12-02T22:40:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 923384832 bytes 2023-12-02T22:40:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 947952640 bytes 2023-12-02T22:40:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 980363264 bytes 2023-12-02T22:40:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1005114368 bytes 2023-12-02T22:40:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1028874240 bytes 2023-12-02T22:40:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1099327488 bytes 2023-12-02T22:40:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1124094976 bytes 2023-12-02T22:40:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1207913472 bytes 2023-12-02T22:40:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1286797312 bytes 2023-12-02T22:40:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1349001216 bytes 2023-12-02T22:40:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1384205312 bytes 2023-12-02T22:40:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1430782976 bytes 2023-12-02T22:40:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1489008640 bytes 2023-12-02T22:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1543003136 bytes 2023-12-02T22:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1602919424 bytes 2023-12-02T22:40:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1641174016 bytes 2023-12-02T22:40:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1706842112 bytes 2023-12-02T22:40:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1751593984 bytes 2023-12-02T22:40:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1833750528 bytes 2023-12-02T22:40:42.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1877436416 bytes 2023-12-02T22:40:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1883086848 bytes 2023-12-02T22:40:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1930675200 bytes 2023-12-02T22:40:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1996663808 bytes 2023-12-02T22:40:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2039255040 bytes 2023-12-02T22:40:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2115604480 bytes 2023-12-02T22:40:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2178014208 bytes 2023-12-02T22:40:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2221547520 bytes 2023-12-02T22:40:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2297681920 bytes 2023-12-02T22:40:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2330555392 bytes 2023-12-02T22:40:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2387763200 bytes 2023-12-02T22:40:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2421322752 bytes 2023-12-02T22:40:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2435485696 bytes 2023-12-02T22:40:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2460744704 bytes 2023-12-02T22:40:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2542084096 bytes 2023-12-02T22:40:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2565279744 bytes 2023-12-02T22:40:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2604232704 bytes 2023-12-02T22:40:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2626381824 bytes 2023-12-02T22:40:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-12-02T22:40:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-02T22:40:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-02T22:40:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_48': 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 14a084bc17ad 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.14a084bc17ad 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:44 2023 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:44 2023 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:44 2023 2023-12-02T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e848264b-794b-4557-91db-d8f1e8b231f2 2023-12-02T22:40:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-02T22:40:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:40:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:40:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 72781824 bytes 2023-12-02T22:40:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 102770688 bytes 2023-12-02T22:40:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 135469056 bytes 2023-12-02T22:40:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 136054784 bytes 2023-12-02T22:40:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 179388416 bytes 2023-12-02T22:40:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 195133440 bytes 2023-12-02T22:40:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 205690880 bytes 2023-12-02T22:40:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 250962944 bytes 2023-12-02T22:40:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 269520896 bytes 2023-12-02T22:40:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 292298752 bytes 2023-12-02T22:40:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 325949440 bytes 2023-12-02T22:40:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 366581760 bytes 2023-12-02T22:40:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 367209472 bytes 2023-12-02T22:40:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 391109632 bytes 2023-12-02T22:40:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 452127744 bytes 2023-12-02T22:40:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 464398336 bytes 2023-12-02T22:40:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 478658560 bytes 2023-12-02T22:40:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 508580864 bytes 2023-12-02T22:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 540465152 bytes 2023-12-02T22:40:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 601989120 bytes 2023-12-02T22:40:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 684008448 bytes 2023-12-02T22:40:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 713176064 bytes 2023-12-02T22:40:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 732780544 bytes 2023-12-02T22:40:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 790665216 bytes 2023-12-02T22:40:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 843144192 bytes 2023-12-02T22:40:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 866486272 bytes 2023-12-02T22:40:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 885764096 bytes 2023-12-02T22:40:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 914435072 bytes 2023-12-02T22:40:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 941242368 bytes 2023-12-02T22:40:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 996393984 bytes 2023-12-02T22:40:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1135718400 bytes 2023-12-02T22:40:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1155602432 bytes 2023-12-02T22:40:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1202776064 bytes 2023-12-02T22:40:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1250722816 bytes 2023-12-02T22:40:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1334028288 bytes 2023-12-02T22:40:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1377258496 bytes 2023-12-02T22:40:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1415471104 bytes 2023-12-02T22:40:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1437659136 bytes 2023-12-02T22:40:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1497225216 bytes 2023-12-02T22:40:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1505903616 bytes 2023-12-02T22:40:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1528477696 bytes 2023-12-02T22:40:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1578582016 bytes 2023-12-02T22:40:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:40:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-02T22:40:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-02T22:40:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_49': 2023-12-02T22:40:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 14a9686ce638 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.14a9686ce638 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:51 2023 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:51 2023 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:51 2023 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8c19922c-ba85-47c0-866f-53244088c863 2023-12-02T22:40:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-02T22:40:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:40:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:40:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 27860992 bytes 2023-12-02T22:40:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 84049920 bytes 2023-12-02T22:40:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 88336384 bytes 2023-12-02T22:40:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 131682304 bytes 2023-12-02T22:40:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 178810880 bytes 2023-12-02T22:40:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 217862144 bytes 2023-12-02T22:40:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 265825280 bytes 2023-12-02T22:40:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 270529536 bytes 2023-12-02T22:40:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 303522816 bytes 2023-12-02T22:40:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 335479808 bytes 2023-12-02T22:40:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 368314368 bytes 2023-12-02T22:40:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 391213056 bytes 2023-12-02T22:40:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 472299520 bytes 2023-12-02T22:40:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 510789632 bytes 2023-12-02T22:40:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 560053248 bytes 2023-12-02T22:40:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 575752192 bytes 2023-12-02T22:40:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 672100352 bytes 2023-12-02T22:40:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 753245184 bytes 2023-12-02T22:40:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 785547264 bytes 2023-12-02T22:40:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 832667648 bytes 2023-12-02T22:40:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 905205760 bytes 2023-12-02T22:40:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 905811968 bytes 2023-12-02T22:40:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 969607168 bytes 2023-12-02T22:40:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 990876672 bytes 2023-12-02T22:40:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1033211904 bytes 2023-12-02T22:40:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1054919680 bytes 2023-12-02T22:40:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1122294784 bytes 2023-12-02T22:40:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1221022720 bytes 2023-12-02T22:40:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1277196288 bytes 2023-12-02T22:40:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1333728256 bytes 2023-12-02T22:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1348162560 bytes 2023-12-02T22:40:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1396848640 bytes 2023-12-02T22:40:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1511258112 bytes 2023-12-02T22:40:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1530556416 bytes 2023-12-02T22:40:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1596266496 bytes 2023-12-02T22:40:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1679238144 bytes 2023-12-02T22:40:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1794594816 bytes 2023-12-02T22:40:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1878080512 bytes 2023-12-02T22:40:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1919594496 bytes 2023-12-02T22:40:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1973648384 bytes 2023-12-02T22:40:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 2042386432 bytes 2023-12-02T22:40:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 2126120960 bytes 2023-12-02T22:40:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2126873600 bytes 2023-12-02T22:40:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2157180928 bytes 2023-12-02T22:40:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2204931072 bytes 2023-12-02T22:40:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2206974976 bytes 2023-12-02T22:40:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2279960576 bytes 2023-12-02T22:40:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snapshot ']' 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:40:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:40:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2023-12-02T22:40:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2023-12-02T22:40:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:40:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' image = image ']' 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:40:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:40:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-02T22:40:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:40:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_50': 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 14b15334a313 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.14b15334a313 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:40:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:59 2023 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:59 2023 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:59 2023 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4b794018-5af4-4f87-a8d7-86141ee6bc32 2023-12-02T22:40:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2023-12-02T22:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-02T22:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:40:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:40:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:40:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:40:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 81803264 bytes 2023-12-02T22:41:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 97714176 bytes 2023-12-02T22:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 160828416 bytes 2023-12-02T22:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 199905280 bytes 2023-12-02T22:41:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 231740416 bytes 2023-12-02T22:41:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 255141888 bytes 2023-12-02T22:41:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 256901120 bytes 2023-12-02T22:41:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 328737792 bytes 2023-12-02T22:41:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 450494464 bytes 2023-12-02T22:41:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 465093632 bytes 2023-12-02T22:41:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 505438208 bytes 2023-12-02T22:41:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 549184512 bytes 2023-12-02T22:41:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 676052992 bytes 2023-12-02T22:41:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 734423040 bytes 2023-12-02T22:41:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 779297792 bytes 2023-12-02T22:41:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 850332672 bytes 2023-12-02T22:41:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 855773184 bytes 2023-12-02T22:41:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 913431552 bytes 2023-12-02T22:41:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 976421888 bytes 2023-12-02T22:41:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 1072201728 bytes 2023-12-02T22:41:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1128840192 bytes 2023-12-02T22:41:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1202616320 bytes 2023-12-02T22:41:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1271111680 bytes 2023-12-02T22:41:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1293830144 bytes 2023-12-02T22:41:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1294444544 bytes 2023-12-02T22:41:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1313498112 bytes 2023-12-02T22:41:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1338281984 bytes 2023-12-02T22:41:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1338902528 bytes 2023-12-02T22:41:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1384013824 bytes 2023-12-02T22:41:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1459599360 bytes 2023-12-02T22:41:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1506220032 bytes 2023-12-02T22:41:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1566466048 bytes 2023-12-02T22:41:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1616463872 bytes 2023-12-02T22:41:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1625191424 bytes 2023-12-02T22:41:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1632021504 bytes 2023-12-02T22:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1687201792 bytes 2023-12-02T22:41:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1780850688 bytes 2023-12-02T22:41:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1835998208 bytes 2023-12-02T22:41:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1838546944 bytes 2023-12-02T22:41:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1859556352 bytes 2023-12-02T22:41:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1905382400 bytes 2023-12-02T22:41:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1981593600 bytes 2023-12-02T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2043419648 bytes 2023-12-02T22:41:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2050385920 bytes 2023-12-02T22:41:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2079137792 bytes 2023-12-02T22:41:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2095162368 bytes 2023-12-02T22:41:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2107288576 bytes 2023-12-02T22:41:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2164908032 bytes 2023-12-02T22:41:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2245231616 bytes 2023-12-02T22:41:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2262075392 bytes 2023-12-02T22:41:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2290099200 bytes 2023-12-02T22:41:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2321030144 bytes 2023-12-02T22:41:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2330058752 bytes 2023-12-02T22:41:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2373122048 bytes 2023-12-02T22:41:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2409989120 bytes 2023-12-02T22:41:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2472367104 bytes 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_count=50 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local count 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_count=0 2023-12-02T22:41:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2023-12-02T22:41:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:41:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2023-12-02T22:41:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-02T22:41:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'images: ' 2023-12-02T22:41:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 2 2023-12-02T22:41:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ count=50 2023-12-02T22:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 50 = 50 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state 2023-12-02T22:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2023-12-02T22:41:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:41:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2023-12-02T22:41:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-02T22:41:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2023-12-02T22:41:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2023-12-02T22:41:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:41:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=OK 2023-12-02T22:41:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = ERROR 2023-12-02T22:41:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = OK 2023-12-02T22:41:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-02T22:41:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_count=50 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local count 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_count=0 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2023-12-02T22:41:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:41:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2023-12-02T22:41:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T22:41:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'images: ' 2023-12-02T22:41:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 2 2023-12-02T22:41:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:41:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ count=50 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 50 = 50 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state 2023-12-02T22:41:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2023-12-02T22:41:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:41:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2023-12-02T22:41:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T22:41:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2023-12-02T22:41:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2023-12-02T22:41:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:41:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=WARNING 2023-12-02T22:41:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test WARNING = ERROR 2023-12-02T22:41:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test WARNING = OK 2023-12-02T22:41:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:41:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 -ne 1 2023-12-02T22:41:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 30 2023-12-02T22:41:17.544 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:41:17.543+0000 7eff3a87e640 -1 reset not still connected to 0x55970e748dd0 2023-12-02T22:41:17.545 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:41:17.543+0000 7eff3a87e640 -1 reset not still connected to 0x5597115f3ba0 2023-12-02T22:41:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T22:41:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2023-12-02T22:41:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2023-12-02T22:41:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=OK 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = ERROR 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = OK 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: compare many images' 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:41:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:41:37 TEST: compare many images 2023-12-02T22:41:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:41:37 TEST: compare many images 2023-12-02T22:41:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:41:39.309 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.26263867765689086) > weight (0.25) 2023-12-02T22:41:39.309 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.43771205702507465) > weight (0.25) 2023-12-02T22:41:39.309 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T22:41:39.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T22:41:39.310 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-02T22:41:39.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:41:39.309+0000 7ffacc581640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 92236) UID: 1000 2023-12-02T22:41:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-02T22:41:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:41:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:41:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-02T22:41:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:41:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:41:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:41:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:41:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:41:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:41:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:41:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:41:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:41:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:41:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:41:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-02T22:41:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 1 closest matches: 2023-12-02T22:41:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_14 2023-12-02T22:41:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:41:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:41:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:41:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:41:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:41:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:41:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:41:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2023-12-02T22:41:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:41:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:41:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:41:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:41:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:41:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:41:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:41:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-02T22:41:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:41:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_1.status 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_1.status 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:41:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.status 2023-12-02T22:41:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_1.status 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_1.status 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-02T22:41:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 119 2023-12-02T22:41:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.status 2023-12-02T22:41:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.status 2023-12-02T22:41:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-02T22:41:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:41:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=119 2023-12-02T22:41:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:41:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T22:41:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:41:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:41:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:41:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:41:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 119 = '' 2023-12-02T22:41:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T22:41:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 119 = '' 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:41:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.status 2023-12-02T22:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T22:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=119 2023-12-02T22:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 119 = 119 2023-12-02T22:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-02T22:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:41:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_1': 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1135b1f487ea 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1135b1f487ea 2023-12-02T22:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:27 2023 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:27 2023 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:27 2023 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 1b41b123-8764-4c2a-808d-b6f2a05a0f17 2023-12-02T22:41:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.export 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.export 2023-12-02T22:41:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.export 2023-12-02T22:41:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.export 2023-12-02T22:41:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:41:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.export 2023-12-02T22:41:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:41:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.export 2023-12-02T22:41:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_1.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_1.export 2023-12-02T22:41:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:41:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2023-12-02T22:41:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-02T22:41:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:41:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:41:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:41:47.311 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T22:41:47.311 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:41:47.311 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T22:41:47.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T22:41:47.311 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T22:41:47.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T22:41:47.314 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2023-12-02T22:41:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-02T22:41:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:41:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:41:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:41:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:41:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:41:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:41:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:41:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:41:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:41:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:41:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:41:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:41:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:41:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:41:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:41:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:41:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-02T22:41:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:41:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:41:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:41:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:41:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:41:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:41:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_2.status 2023-12-02T22:41:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_2.status 2023-12-02T22:41:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.status 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_2.status 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_2.status 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-02T22:41:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 121 2023-12-02T22:41:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.status 2023-12-02T22:41:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.status 2023-12-02T22:41:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-02T22:41:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:41:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=121 2023-12-02T22:41:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:41:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-02T22:41:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:41:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:41:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:41:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:41:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 121 = '' 2023-12-02T22:41:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:41:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-02T22:41:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:41:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:41:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:41:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:41:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 121 = '' 2023-12-02T22:41:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:41:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.status 2023-12-02T22:41:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-02T22:41:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=121 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 121 = 121 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:41:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:41:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_2': 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 11865f5a0353 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.11865f5a0353 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:41:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:41 2023 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:41 2023 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:41 2023 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f793e7a7-0001-44a2-82ae-444e4e65879f 2023-12-02T22:41:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.export 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.export 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.export 2023-12-02T22:41:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.export 2023-12-02T22:41:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:41:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.export 2023-12-02T22:41:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:41:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.export 2023-12-02T22:41:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_2.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_2.export 2023-12-02T22:41:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:41:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2023-12-02T22:41:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-02T22:41:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:41:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:41:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:41:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:41:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:41:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:41:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:41:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:41:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:41:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-02T22:41:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:41:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_34 2023-12-02T22:41:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_38 2023-12-02T22:41:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_39 2023-12-02T22:41:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:41:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:41:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:41:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:41:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:41:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:41:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:41:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:41:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:41:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-12-02T22:41:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:41:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_32 2023-12-02T22:41:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_33 2023-12-02T22:41:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_36 2023-12-02T22:41:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:41:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:41:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:41:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:41:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-12-02T22:41:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 2 closest matches: 2023-12-02T22:41:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_31 2023-12-02T22:41:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_35 2023-12-02T22:41:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:41:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:41:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:41:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:41:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:41:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:41:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:41:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:41:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2023-12-02T22:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-02T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_3.status 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_3.status 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.status 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_3.status 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_3.status 2023-12-02T22:41:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:41:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-02T22:41:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-02T22:41:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 123 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.status 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.status 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-02T22:41:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=123 2023-12-02T22:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T22:41:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=70 2023-12-02T22:41:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 123 = 70 2023-12-02T22:41:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:41:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T22:41:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 123 = '' 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:41:56.207 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T22:41:56.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T22:41:56.208 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T22:41:56.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:41:56.208+0000 7fc1b4686640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 102860) UID: 1000 2023-12-02T22:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T22:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:41:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:41:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:41:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:41:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 123 = '' 2023-12-02T22:41:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:41:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.status 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T22:41:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:41:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=123 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 123 = 123 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:41:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:41:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:41:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_3': 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1201814f2c80 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1201814f2c80 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:33:56 2023 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:33:56 2023 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:33:56 2023 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c028f4b4-ce10-4094-9ec7-655b7f816feb 2023-12-02T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:41:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.export 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.export 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.export 2023-12-02T22:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.export 2023-12-02T22:41:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:41:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.export 2023-12-02T22:41:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:41:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.export 2023-12-02T22:41:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_3.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_3.export 2023-12-02T22:41:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:41:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:41:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2023-12-02T22:41:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-02T22:41:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:41:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:41:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:41:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:41:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:42:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:42:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:42:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:42:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:42:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T22:42:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T22:42:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 1 closest matches: 2023-12-02T22:42:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_43 2023-12-02T22:42:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:42:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:42:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:42:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T22:42:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:42:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_41 2023-12-02T22:42:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_42 2023-12-02T22:42:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_49 2023-12-02T22:42:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T22:42:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:42:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:42:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:42:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:42:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T22:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 4 closest matches: 2023-12-02T22:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_45 2023-12-02T22:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_46 2023-12-02T22:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_47 2023-12-02T22:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_48 2023-12-02T22:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 1 2023-12-02T22:42:03.209 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T22:42:03.209 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:42:03.209 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T22:42:03.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T22:42:03.210 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T22:42:03.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T22:42:03.212 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-12-02T22:42:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T22:42:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T22:42:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T22:42:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T22:42:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 1 closest matches: 2023-12-02T22:42:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_43 2023-12-02T22:42:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T22:42:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:42:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:42:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:42:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T22:42:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:42:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_41 2023-12-02T22:42:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_42 2023-12-02T22:42:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_49 2023-12-02T22:42:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T22:42:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 4 closest matches: 2023-12-02T22:42:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_45 2023-12-02T22:42:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_46 2023-12-02T22:42:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_47 2023-12-02T22:42:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_48 2023-12-02T22:42:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 2 2023-12-02T22:42:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T22:42:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T22:42:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T22:42:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T22:42:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 1 closest matches: 2023-12-02T22:42:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_43 2023-12-02T22:42:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:42:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:42:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T22:42:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:42:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_41 2023-12-02T22:42:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_42 2023-12-02T22:42:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_49 2023-12-02T22:42:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:42:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:42:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T22:42:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 4 closest matches: 2023-12-02T22:42:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_45 2023-12-02T22:42:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_46 2023-12-02T22:42:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_47 2023-12-02T22:42:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_48 2023-12-02T22:42:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 4 2023-12-02T22:42:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T22:42:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T22:42:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T22:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 1 closest matches: 2023-12-02T22:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_43 2023-12-02T22:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T22:42:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:42:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:42:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:42:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:42:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T22:42:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:42:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_41 2023-12-02T22:42:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_42 2023-12-02T22:42:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_49 2023-12-02T22:42:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T22:42:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:42:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:42:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:42:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T22:42:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 4 closest matches: 2023-12-02T22:42:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_45 2023-12-02T22:42:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_46 2023-12-02T22:42:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_47 2023-12-02T22:42:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_48 2023-12-02T22:42:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 8 2023-12-02T22:42:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T22:42:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T22:42:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T22:42:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 1 closest matches: 2023-12-02T22:42:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_43 2023-12-02T22:42:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:42:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:42:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:42:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T22:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 3 closest matches: 2023-12-02T22:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_41 2023-12-02T22:42:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_42 2023-12-02T22:42:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_49 2023-12-02T22:42:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:42:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:42:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:42:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:42:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T22:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 4 closest matches: 2023-12-02T22:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_45 2023-12-02T22:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_46 2023-12-02T22:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_47 2023-12-02T22:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_48 2023-12-02T22:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 8 2023-12-02T22:42:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T22:42:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T22:42:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T22:42:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:42:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:42:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:42:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:42:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:42:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:42:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_4.status 2023-12-02T22:42:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_4.status 2023-12-02T22:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.status 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_4.status 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_4.status 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-02T22:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-02T22:42:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 125 2023-12-02T22:42:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.status 2023-12-02T22:42:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.status 2023-12-02T22:42:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-02T22:42:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:42:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=125 2023-12-02T22:42:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-02T22:42:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 125 = '' 2023-12-02T22:42:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-02T22:42:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 125 = '' 2023-12-02T22:42:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.status 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-02T22:42:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=125 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 125 = 125 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:42:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-02T22:42:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_4': 2023-12-02T22:42:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:42:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127bc1ff1593 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127bc1ff1593 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:01 2023 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:01 2023 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:01 2023 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a8f37621-7a4e-4f6c-b01d-f20a1d76f088 2023-12-02T22:42:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:42:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-02T22:42:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:42:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:42:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.export 2023-12-02T22:42:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.export 2023-12-02T22:42:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.export 2023-12-02T22:42:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.export 2023-12-02T22:42:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.export 2023-12-02T22:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.export 2023-12-02T22:42:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_4.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_4.export 2023-12-02T22:42:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:42:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:42:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-02T22:42:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-02T22:42:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:42:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:42:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:42:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:42:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_5.status 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_5.status 2023-12-02T22:42:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.status 2023-12-02T22:42:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_5.status 2023-12-02T22:42:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_5.status 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-02T22:42:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 127 2023-12-02T22:42:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.status 2023-12-02T22:42:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.status 2023-12-02T22:42:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-02T22:42:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:42:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=127 2023-12-02T22:42:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T22:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=72 2023-12-02T22:42:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 127 = 72 2023-12-02T22:42:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T22:42:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 127 = '' 2023-12-02T22:42:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:42:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T22:42:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 127 = '' 2023-12-02T22:42:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:42:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.status 2023-12-02T22:42:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T22:42:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=127 2023-12-02T22:42:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 127 = 127 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:42:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-02T22:42:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_5': 2023-12-02T22:42:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:42:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12987da8ea1 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12987da8ea1 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:12 2023 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:12 2023 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:12 2023 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 3083f507-4484-4113-938a-b4257c974175 2023-12-02T22:42:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:42:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-02T22:42:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:42:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:42:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.export 2023-12-02T22:42:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.export 2023-12-02T22:42:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.export 2023-12-02T22:42:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.export 2023-12-02T22:42:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.export 2023-12-02T22:42:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.export 2023-12-02T22:42:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_5.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_5.export 2023-12-02T22:42:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-02T22:42:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:42:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:42:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:42:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:42:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:42:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-02T22:42:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-02T22:42:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-12-02T22:42:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-12-02T22:42:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:42:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:42:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:42:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:42:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_6.status 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_6.status 2023-12-02T22:42:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.status 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_6.status 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_6.status 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-02T22:42:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 129 2023-12-02T22:42:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.status 2023-12-02T22:42:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.status 2023-12-02T22:42:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-02T22:42:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:42:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=129 2023-12-02T22:42:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-02T22:42:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 129 = '' 2023-12-02T22:42:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:42:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-02T22:42:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 129 = '' 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.status 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-02T22:42:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=129 2023-12-02T22:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 129 = 129 2023-12-02T22:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-02T22:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:42:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-02T22:42:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_6': 2023-12-02T22:42:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:42:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:42:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 113535f8f8b9 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.113535f8f8b9 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:19 2023 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:19 2023 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:19 2023 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 6f4a39a9-03a2-4b4f-a1b0-8e15b573f1ab 2023-12-02T22:42:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.export 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.export 2023-12-02T22:42:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.export 2023-12-02T22:42:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.export 2023-12-02T22:42:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.export 2023-12-02T22:42:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.export 2023-12-02T22:42:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_6.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_6.export 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:42:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:42:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-02T22:42:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:42:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-02T22:42:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-02T22:42:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:42:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:42:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:42:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:42:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:42:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-02T22:42:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-02T22:42:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:42:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:42:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:42:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:42:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:42:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:42:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:42:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-12-02T22:42:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-02T22:42:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:42:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:42:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:42:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:42:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:42:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2023-12-02T22:42:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:42:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:42:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:42:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:42:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:42:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_7.status 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_7.status 2023-12-02T22:42:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.status 2023-12-02T22:42:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_7.status 2023-12-02T22:42:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_7.status 2023-12-02T22:42:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-02T22:42:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-02T22:42:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 131 2023-12-02T22:42:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.status 2023-12-02T22:42:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.status 2023-12-02T22:42:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-02T22:42:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:42:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=131 2023-12-02T22:42:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=74 2023-12-02T22:42:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 131 = 74 2023-12-02T22:42:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:42:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T22:42:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 131 = '' 2023-12-02T22:42:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T22:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:42:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:42:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:42:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:42:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 131 = '' 2023-12-02T22:42:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.status 2023-12-02T22:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T22:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:42:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=131 2023-12-02T22:42:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 131 = 131 2023-12-02T22:42:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-02T22:42:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:42:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-02T22:42:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_7': 2023-12-02T22:42:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 118683629c4f 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.118683629c4f 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:25 2023 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:25 2023 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:25 2023 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:42:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 29d0f358-af7a-4fe0-84a4-8618fbdbda52 2023-12-02T22:42:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:42:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-02T22:42:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:42:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:42:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.export 2023-12-02T22:42:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.export 2023-12-02T22:42:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.export 2023-12-02T22:42:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.export 2023-12-02T22:42:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.export 2023-12-02T22:42:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:42:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.export 2023-12-02T22:42:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_7.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_7.export 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:42:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:42:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-02T22:42:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:42:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-02T22:42:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:42:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:42:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:42:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:42:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:42:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:42:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:42:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:42:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-02T22:42:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:42:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:42:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:42:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:42:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:42:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:42:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:42:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:42:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-02T22:42:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_8.status 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_8.status 2023-12-02T22:42:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.status 2023-12-02T22:42:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_8.status 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_8.status 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-02T22:42:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 133 2023-12-02T22:42:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.status 2023-12-02T22:42:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:42:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.status 2023-12-02T22:42:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-02T22:42:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:42:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:42:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=133 2023-12-02T22:42:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:42:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:42:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-02T22:42:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 133 = '' 2023-12-02T22:43:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-02T22:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 133 = '' 2023-12-02T22:43:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.status 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-02T22:43:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=133 2023-12-02T22:43:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 133 = 133 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_8': 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127b6f6dccb0 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127b6f6dccb0 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:30 2023 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:30 2023 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:30 2023 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 54d6e2fd-a8c4-41f2-93f4-8602f730be4d 2023-12-02T22:43:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.export 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.export 2023-12-02T22:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.export 2023-12-02T22:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.export 2023-12-02T22:43:02.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.export 2023-12-02T22:43:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.export 2023-12-02T22:43:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_8.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_8.export 2023-12-02T22:43:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2023-12-02T22:43:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-02T22:43:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-02T22:43:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-02T22:43:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-02T22:43:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-02T22:43:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-02T22:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_9.status 2023-12-02T22:43:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_9.status 2023-12-02T22:43:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.status 2023-12-02T22:43:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_9.status 2023-12-02T22:43:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_9.status 2023-12-02T22:43:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-02T22:43:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-02T22:43:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 135 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.status 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.status 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-02T22:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=135 2023-12-02T22:43:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-02T22:43:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 135 = '' 2023-12-02T22:43:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-02T22:43:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 135 = '' 2023-12-02T22:43:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.status 2023-12-02T22:43:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-02T22:43:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=135 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 135 = 135 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_9': 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1135f88042ea 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1135f88042ea 2023-12-02T22:43:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:43 2023 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:43 2023 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:43 2023 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f162a3b0-3c88-4e6c-ae84-ffb10c73c491 2023-12-02T22:43:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.export 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.export 2023-12-02T22:43:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.export 2023-12-02T22:43:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.export 2023-12-02T22:43:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.export 2023-12-02T22:43:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.export 2023-12-02T22:43:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_9.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_9.export 2023-12-02T22:43:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2023-12-02T22:43:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-02T22:43:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-02T22:43:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-02T22:43:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-02T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-02T22:43:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-02T22:43:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-02T22:43:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:43:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:43:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:43:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:43:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-02T22:43:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-02T22:43:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-12-02T22:43:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:43:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:43:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:43:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-12-02T22:43:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-02T22:43:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-12-02T22:43:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:43:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:43:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:43:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:43:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:43:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:43:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-12-02T22:43:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-02T22:43:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_10.status 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_10.status 2023-12-02T22:43:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.status 2023-12-02T22:43:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_10.status 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_10.status 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-02T22:43:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 137 2023-12-02T22:43:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.status 2023-12-02T22:43:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.status 2023-12-02T22:43:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-02T22:43:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=137 2023-12-02T22:43:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T22:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 137 = '' 2023-12-02T22:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T22:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 137 = '' 2023-12-02T22:43:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.status 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T22:43:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=137 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 137 = 137 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-02T22:43:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_10': 2023-12-02T22:43:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 11868ebd7e0b 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.11868ebd7e0b 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:49 2023 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:49 2023 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:49 2023 2023-12-02T22:43:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 744cfe14-46e0-4471-aa62-cd6ed74706ca 2023-12-02T22:43:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.export 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.export 2023-12-02T22:43:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.export 2023-12-02T22:43:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.export 2023-12-02T22:43:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.export 2023-12-02T22:43:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.export 2023-12-02T22:43:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_10.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_10.export 2023-12-02T22:43:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2023-12-02T22:43:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-02T22:43:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-02T22:43:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-02T22:43:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:43:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:43:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-02T22:43:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-02T22:43:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:43:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:43:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:43:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-12-02T22:43:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_11.status 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_11.status 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.status 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_11.status 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_11.status 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-02T22:43:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 139 2023-12-02T22:43:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.status 2023-12-02T22:43:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.status 2023-12-02T22:43:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-02T22:43:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=139 2023-12-02T22:43:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T22:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=78 2023-12-02T22:43:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 139 = 78 2023-12-02T22:43:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T22:43:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 139 = '' 2023-12-02T22:43:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T22:43:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 139 = '' 2023-12-02T22:43:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:43:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.status 2023-12-02T22:43:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T22:43:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=139 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 139 = 139 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-02T22:43:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_11': 2023-12-02T22:43:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127b53cc05a1 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127b53cc05a1 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:34:55 2023 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:34:55 2023 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:34:55 2023 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7d477ce5-b8e7-48b3-aa17-29ac3ee8f70c 2023-12-02T22:43:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.export 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.export 2023-12-02T22:43:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.export 2023-12-02T22:43:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.export 2023-12-02T22:43:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.export 2023-12-02T22:43:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.export 2023-12-02T22:43:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_11.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_11.export 2023-12-02T22:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-02T22:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-02T22:43:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-02T22:43:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-02T22:43:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-02T22:43:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-02T22:43:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:43:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:43:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:43:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:43:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-02T22:43:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-02T22:43:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-02T22:43:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:43:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:43:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:43:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-02T22:43:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-02T22:43:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:43:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:43:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:43:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:43:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:43:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-02T22:43:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_12.status 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_12.status 2023-12-02T22:43:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.status 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_12.status 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_12.status 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-02T22:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 141 2023-12-02T22:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.status 2023-12-02T22:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.status 2023-12-02T22:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-02T22:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=141 2023-12-02T22:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-02T22:43:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 141 = '' 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-02T22:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 141 = '' 2023-12-02T22:43:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.status 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-02T22:43:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=141 2023-12-02T22:43:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 141 = 141 2023-12-02T22:43:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-02T22:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_12': 2023-12-02T22:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1298f4111469 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1298f4111469 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:01 2023 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:01 2023 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:01 2023 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 87b8efc0-f045-4058-89b2-d6580e15413a 2023-12-02T22:43:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-02T22:43:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:43:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.export 2023-12-02T22:43:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.export 2023-12-02T22:43:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.export 2023-12-02T22:43:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.export 2023-12-02T22:43:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.export 2023-12-02T22:43:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.export 2023-12-02T22:43:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_12.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_12.export 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-02T22:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-02T22:43:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-02T22:43:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-02T22:43:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-02T22:43:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_13.status 2023-12-02T22:43:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_13.status 2023-12-02T22:43:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.status 2023-12-02T22:43:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_13.status 2023-12-02T22:43:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_13.status 2023-12-02T22:43:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-02T22:43:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-02T22:43:36.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.35071383172171033) > weight (0.25) 2023-12-02T22:43:36.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9907328611302719) > weight (0.25) 2023-12-02T22:43:36.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7575426097798414) > weight (0.25) 2023-12-02T22:43:36.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3148091205223036) > weight (0.25) 2023-12-02T22:43:36.315 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-02T22:43:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 143 2023-12-02T22:43:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.status 2023-12-02T22:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.status 2023-12-02T22:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-02T22:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=143 2023-12-02T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T22:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 143 = '' 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T22:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 143 = '' 2023-12-02T22:43:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.status 2023-12-02T22:43:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T22:43:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=143 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 143 = 143 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_13': 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1135638cda3b 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1135638cda3b 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:14 2023 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:14 2023 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:14 2023 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d76b9c8b-eb85-4586-95f4-981ca6150274 2023-12-02T22:43:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.export 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.export 2023-12-02T22:43:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.export 2023-12-02T22:43:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.export 2023-12-02T22:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.export 2023-12-02T22:43:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.export 2023-12-02T22:43:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_13.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_13.export 2023-12-02T22:43:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-02T22:43:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-02T22:43:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_14.status 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_14.status 2023-12-02T22:43:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.status 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_14.status 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_14.status 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-02T22:43:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 145 2023-12-02T22:43:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.status 2023-12-02T22:43:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.status 2023-12-02T22:43:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-02T22:43:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=145 2023-12-02T22:43:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 145 = '' 2023-12-02T22:43:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T22:43:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 145 = '' 2023-12-02T22:43:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.status 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T22:43:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=145 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 145 = 145 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_14': 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1186cd042384 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1186cd042384 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:23 2023 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:23 2023 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:23 2023 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: da7e5eab-bc9e-4019-826f-9d3346e2d46b 2023-12-02T22:43:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.export 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.export 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.export 2023-12-02T22:43:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.export 2023-12-02T22:43:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.export 2023-12-02T22:43:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.export 2023-12-02T22:43:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_14.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_14.export 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-02T22:43:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-02T22:43:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:43:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:43:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:43:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:43:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-12-02T22:43:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-02T22:43:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:43:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:43:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2023-12-02T22:43:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_15.status 2023-12-02T22:43:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_15.status 2023-12-02T22:43:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.status 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_15.status 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_15.status 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-02T22:43:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 147 2023-12-02T22:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.status 2023-12-02T22:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.status 2023-12-02T22:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-02T22:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=147 2023-12-02T22:43:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T22:43:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=82 2023-12-02T22:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 147 = 82 2023-12-02T22:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:43:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T22:43:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 147 = '' 2023-12-02T22:43:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T22:43:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:43:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:43:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:43:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:43:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 147 = '' 2023-12-02T22:43:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:43:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.status 2023-12-02T22:43:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T22:43:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=147 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 147 = 147 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:43:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_15': 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127b2c6355e6 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127b2c6355e6 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:43:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:38 2023 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:38 2023 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:38 2023 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: b7fa7b77-c7c1-4f78-b242-68962c8e8eb8 2023-12-02T22:43:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.export 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.export 2023-12-02T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.export 2023-12-02T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.export 2023-12-02T22:43:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.export 2023-12-02T22:43:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:43:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.export 2023-12-02T22:43:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_15.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_15.export 2023-12-02T22:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:43:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:43:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-02T22:43:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:43:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-02T22:43:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:43:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:43:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-02T22:43:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:43:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:43:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-02T22:43:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:43:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:43:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:43:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:43:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:43:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:43:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:43:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:43:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-02T22:43:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_16.status 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_16.status 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.status 2023-12-02T22:43:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_16.status 2023-12-02T22:43:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_16.status 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-02T22:43:59.213 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T22:43:59.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T22:43:59.214 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T22:43:59.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:43:59.213+0000 7fcadd1c5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 105066) UID: 1000 2023-12-02T22:43:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 149 2023-12-02T22:43:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.status 2023-12-02T22:43:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:43:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.status 2023-12-02T22:43:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-02T22:43:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:43:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:43:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=149 2023-12-02T22:43:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:43:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:43:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:43:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:43:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:43:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:43:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:43:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:43:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:44:00.214 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T22:44:00.214 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:44:00.215 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T22:44:00.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T22:44:00.215 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T22:44:00.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T22:44:00.217 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-02T22:44:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:44:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 4 2023-12-02T22:44:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 4 2023-12-02T22:44:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:44:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=83 2023-12-02T22:44:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 83 2023-12-02T22:44:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:44:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.status 2023-12-02T22:44:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T22:44:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=149 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 149 = 149 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:44:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_16': 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12981cdeaf96 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12981cdeaf96 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:44:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:35:45 2023 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:35:45 2023 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:35:45 2023 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 99280131-88b8-4f83-930d-440a9d1456ee 2023-12-02T22:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.export 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.export 2023-12-02T22:44:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.export 2023-12-02T22:44:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.export 2023-12-02T22:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.export 2023-12-02T22:44:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.export 2023-12-02T22:44:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_16.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_16.export 2023-12-02T22:44:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:44:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2023-12-02T22:44:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-02T22:44:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:44:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-02T22:44:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-02T22:44:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:44:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:44:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:44:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:44:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:44:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-02T22:44:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:44:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:44:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:44:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:44:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_17.status 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_17.status 2023-12-02T22:44:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.status 2023-12-02T22:44:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_17.status 2023-12-02T22:44:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_17.status 2023-12-02T22:44:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-02T22:44:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-02T22:44:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 151 2023-12-02T22:44:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.status 2023-12-02T22:44:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.status 2023-12-02T22:44:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-02T22:44:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:44:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=151 2023-12-02T22:44:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:44:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-02T22:44:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:44:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:44:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:44:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:44:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 151 = '' 2023-12-02T22:44:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:44:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-02T22:44:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:44:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:44:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:44:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:44:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 151 = '' 2023-12-02T22:44:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:44:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.status 2023-12-02T22:44:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-02T22:44:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=151 2023-12-02T22:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 151 = 151 2023-12-02T22:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-02T22:44:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_17': 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1e9f92e35 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1e9f92e35 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:26 2023 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:26 2023 2023-12-02T22:44:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:26 2023 2023-12-02T22:44:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:44:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:44:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:44:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 06d1cdd3-95e7-4459-aa25-f7cf37c99105 2023-12-02T22:44:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:44:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-02T22:44:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:44:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:44:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.export 2023-12-02T22:44:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.export 2023-12-02T22:44:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.export 2023-12-02T22:44:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.export 2023-12-02T22:44:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.export 2023-12-02T22:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.export 2023-12-02T22:44:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_17.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_17.export 2023-12-02T22:44:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:44:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2023-12-02T22:44:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-02T22:44:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:44:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:44:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-02T22:44:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-02T22:44:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:44:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:44:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:44:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:44:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:44:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-02T22:44:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-02T22:44:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:44:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:44:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:44:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:44:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:44:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-12-02T22:44:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-02T22:44:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:44:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:44:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:44:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:44:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:44:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:44:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:44:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:44:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:44:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:44:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2023-12-02T22:44:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:44:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:44:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:44:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:44:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:44:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:44:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:44:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-02T22:44:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_18.status 2023-12-02T22:44:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_18.status 2023-12-02T22:44:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.status 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_18.status 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_18.status 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-02T22:44:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-02T22:44:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 153 2023-12-02T22:44:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.status 2023-12-02T22:44:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.status 2023-12-02T22:44:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-02T22:44:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:44:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=153 2023-12-02T22:44:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:44:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T22:44:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=85 2023-12-02T22:44:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 153 = 85 2023-12-02T22:44:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:44:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T22:44:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 153 = '' 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:44:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T22:44:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:44:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:44:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:44:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:44:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 153 = '' 2023-12-02T22:44:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:44:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.status 2023-12-02T22:44:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T22:44:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=153 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 153 = 153 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:44:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-02T22:44:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_18': 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 118625ec05fc 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.118625ec05fc 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:00 2023 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:00 2023 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:00 2023 2023-12-02T22:44:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:44:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:44:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:44:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: ebdd1cd5-86de-4c0d-bc2d-c273a9a23947 2023-12-02T22:44:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:44:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-02T22:44:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:44:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:44:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.export 2023-12-02T22:44:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.export 2023-12-02T22:44:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.export 2023-12-02T22:44:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.export 2023-12-02T22:44:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.export 2023-12-02T22:44:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.export 2023-12-02T22:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_18.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_18.export 2023-12-02T22:44:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:44:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:44:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-02T22:44:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:44:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:44:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:44:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-02T22:44:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-02T22:44:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:44:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:44:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:44:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:44:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:44:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:44:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:44:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-02T22:44:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:44:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:44:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:44:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:44:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:44:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_19.status 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_19.status 2023-12-02T22:44:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.status 2023-12-02T22:44:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_19.status 2023-12-02T22:44:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_19.status 2023-12-02T22:44:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-02T22:44:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-02T22:44:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 155 2023-12-02T22:44:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.status 2023-12-02T22:44:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.status 2023-12-02T22:44:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-02T22:44:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:44:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=155 2023-12-02T22:44:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T22:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:44:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:44:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:44:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:44:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 155 = '' 2023-12-02T22:44:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:44:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T22:44:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:44:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:44:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:44:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:44:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 155 = '' 2023-12-02T22:44:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.status 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T22:44:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=155 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 155 = 155 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:44:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-02T22:44:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_19': 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1cce8983c 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1cce8983c 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:12 2023 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:12 2023 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:12 2023 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:44:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:44:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:44:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a8c6040f-b031-4199-aca7-736fca431324 2023-12-02T22:44:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:44:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.export 2023-12-02T22:44:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.export 2023-12-02T22:44:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.export 2023-12-02T22:44:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.export 2023-12-02T22:44:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.export 2023-12-02T22:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:44:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.export 2023-12-02T22:44:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_19.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_19.export 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:44:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:44:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-02T22:44:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:44:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-02T22:44:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:44:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:44:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:44:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:44:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-02T22:44:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:44:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:44:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:44:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:44:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-02T22:44:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-02T22:44:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:44:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:44:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:44:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:44:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:44:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:44:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:44:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:44:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-02T22:44:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:44:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:44:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:44:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:44:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:44:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:44:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:44:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:44:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_20.status 2023-12-02T22:44:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_20.status 2023-12-02T22:44:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.status 2023-12-02T22:44:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_20.status 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_20.status 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-02T22:44:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 157 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.status 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.status 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-02T22:44:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:44:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:44:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=157 2023-12-02T22:44:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:44:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:44:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:45:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:45:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:45:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T22:45:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 157 = '' 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:45:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:45:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:45:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T22:45:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 157 = '' 2023-12-02T22:45:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.status 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:01.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8113951416621301) > weight (0.25) 2023-12-02T22:45:01.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4394504222726001) > weight (0.25) 2023-12-02T22:45:01.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.32993115799151407) > weight (0.25) 2023-12-02T22:45:01.317 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-02T22:45:01.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-02T22:45:01.317 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:45:01.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:45:01.316+0000 7f0c45a8d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 101652) UID: 1000 2023-12-02T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=157 2023-12-02T22:45:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 157 = 157 2023-12-02T22:45:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_20': 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1c98d9186 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1c98d9186 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:20 2023 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:20 2023 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:20 2023 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 89f1b9c1-adde-4973-967a-efbd5a93afd6 2023-12-02T22:45:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-02T22:45:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:45:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.export 2023-12-02T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.export 2023-12-02T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.export 2023-12-02T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.export 2023-12-02T22:45:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.export 2023-12-02T22:45:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.export 2023-12-02T22:45:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_20.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_20.export 2023-12-02T22:45:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-02T22:45:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-02T22:45:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-02T22:45:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-02T22:45:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-02T22:45:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:45:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:45:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:45:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-12-02T22:45:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-02T22:45:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:45:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:45:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:45:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-12-02T22:45:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_21.status 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_21.status 2023-12-02T22:45:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.status 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_21.status 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_21.status 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-02T22:45:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 159 2023-12-02T22:45:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.status 2023-12-02T22:45:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.status 2023-12-02T22:45:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-02T22:45:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=159 2023-12-02T22:45:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-02T22:45:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 159 = '' 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-02T22:45:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 159 = '' 2023-12-02T22:45:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:07.984 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:45:07.982+0000 7f315ad82640 -1 reset not still connected to 0x564e0e56bba0 2023-12-02T22:45:07.984 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:45:07.982+0000 7f315ad82640 -1 reset not still connected to 0x564e199429c0 2023-12-02T22:45:07.984 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:45:07.982+0000 7f315ad82640 -1 reset not still connected to 0x564e1b8b3e10 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.status 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-02T22:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=159 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 159 = 159 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-02T22:45:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_21': 2023-12-02T22:45:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 118696715d04 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.118696715d04 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:35 2023 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:35 2023 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:35 2023 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 758205de-a838-479c-8d08-b70c14a4aacc 2023-12-02T22:45:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.export 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.export 2023-12-02T22:45:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.export 2023-12-02T22:45:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.export 2023-12-02T22:45:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.export 2023-12-02T22:45:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.export 2023-12-02T22:45:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_21.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_21.export 2023-12-02T22:45:10.318 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-02T22:45:10.319 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:45:10.319 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-02T22:45:10.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T22:45:10.319 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T22:45:10.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T22:45:10.322 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-02T22:45:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-02T22:45:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-02T22:45:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-02T22:45:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_22.status 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_22.status 2023-12-02T22:45:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.status 2023-12-02T22:45:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_22.status 2023-12-02T22:45:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_22.status 2023-12-02T22:45:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-02T22:45:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-02T22:45:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 161 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.status 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.status 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-02T22:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=161 2023-12-02T22:45:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T22:45:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=89 2023-12-02T22:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 161 = 89 2023-12-02T22:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T22:45:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 161 = '' 2023-12-02T22:45:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T22:45:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 161 = '' 2023-12-02T22:45:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:45:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.status 2023-12-02T22:45:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T22:45:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=161 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 161 = 161 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-02T22:45:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_22': 2023-12-02T22:45:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127bf2f5aa6e 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127bf2f5aa6e 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:46 2023 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:46 2023 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:46 2023 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 5661f1b2-001e-4d14-9686-8b906099edb6 2023-12-02T22:45:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-02T22:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.export 2023-12-02T22:45:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.export 2023-12-02T22:45:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.export 2023-12-02T22:45:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.export 2023-12-02T22:45:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.export 2023-12-02T22:45:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.export 2023-12-02T22:45:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_22.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_22.export 2023-12-02T22:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2023-12-02T22:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-02T22:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-02T22:45:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-02T22:45:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-02T22:45:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-02T22:45:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-02T22:45:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:45:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:45:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:45:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:45:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:45:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:45:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-02T22:45:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-02T22:45:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-02T22:45:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:45:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:45:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-02T22:45:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_23.status 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_23.status 2023-12-02T22:45:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.status 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_23.status 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_23.status 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-02T22:45:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 163 2023-12-02T22:45:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.status 2023-12-02T22:45:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.status 2023-12-02T22:45:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-02T22:45:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=163 2023-12-02T22:45:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-02T22:45:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 163 = '' 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-02T22:45:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 163 = '' 2023-12-02T22:45:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.status 2023-12-02T22:45:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-02T22:45:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=163 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 163 = 163 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_23': 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1298952673de 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1298952673de 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:52 2023 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:52 2023 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:52 2023 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 645a0ff2-7b25-4678-9148-2db63c48f624 2023-12-02T22:45:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.export 2023-12-02T22:45:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.export 2023-12-02T22:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.export 2023-12-02T22:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.export 2023-12-02T22:45:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.export 2023-12-02T22:45:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.export 2023-12-02T22:45:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_23.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_23.export 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T22:45:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_24.status 2023-12-02T22:45:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_24.status 2023-12-02T22:45:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.status 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_24.status 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_24.status 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-02T22:45:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 165 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.status 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.status 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-02T22:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=165 2023-12-02T22:45:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-02T22:45:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 165 = '' 2023-12-02T22:45:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-02T22:45:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 165 = '' 2023-12-02T22:45:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.status 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-02T22:45:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=165 2023-12-02T22:45:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 165 = 165 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-02T22:45:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_24': 2023-12-02T22:45:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a18009267a 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a18009267a 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:36:59 2023 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:36:59 2023 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:36:59 2023 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 208a8244-37c7-4cb5-8dbd-872c1e9d06f2 2023-12-02T22:45:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.export 2023-12-02T22:45:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.export 2023-12-02T22:45:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.export 2023-12-02T22:45:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.export 2023-12-02T22:45:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.export 2023-12-02T22:45:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.export 2023-12-02T22:45:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_24.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_24.export 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:35.218 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8116344895485575) > weight (0.25) 2023-12-02T22:45:35.218 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.26431632120802984) > weight (0.25) 2023-12-02T22:45:35.218 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4395180380142927) > weight (0.25) 2023-12-02T22:45:35.219 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4369809240017847) > weight (0.25) 2023-12-02T22:45:35.219 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-02T22:45:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-02T22:45:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:45:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:45:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:45:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:45:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-02T22:45:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-02T22:45:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-12-02T22:45:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:45:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:45:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:45:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:45:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-12-02T22:45:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_25.status 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_25.status 2023-12-02T22:45:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.status 2023-12-02T22:45:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_25.status 2023-12-02T22:45:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_25.status 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-02T22:45:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 167 2023-12-02T22:45:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.status 2023-12-02T22:45:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.status 2023-12-02T22:45:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-02T22:45:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=167 2023-12-02T22:45:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-02T22:45:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 167 = '' 2023-12-02T22:45:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-02T22:45:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 167 = '' 2023-12-02T22:45:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.status 2023-12-02T22:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-02T22:45:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=167 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 167 = 167 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-02T22:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_25': 2023-12-02T22:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 11867d9ca3d3 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.11867d9ca3d3 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:05 2023 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:05 2023 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:05 2023 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c16dc03c-c292-424e-be08-d6e908fda7e8 2023-12-02T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.export 2023-12-02T22:45:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.export 2023-12-02T22:45:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.export 2023-12-02T22:45:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.export 2023-12-02T22:45:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.export 2023-12-02T22:45:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.export 2023-12-02T22:45:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_25.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_25.export 2023-12-02T22:45:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2023-12-02T22:45:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-02T22:45:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-02T22:45:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-02T22:45:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-02T22:45:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-02T22:45:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-02T22:45:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-02T22:45:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:45:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:45:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:45:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:45:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:45:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-02T22:45:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-02T22:45:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:45:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:45:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:45:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:45:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:45:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:45:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-12-02T22:45:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_26.status 2023-12-02T22:45:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_26.status 2023-12-02T22:45:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.status 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_26.status 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_26.status 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-02T22:45:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-02T22:45:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 169 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.status 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.status 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-02T22:45:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=169 2023-12-02T22:45:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-02T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 169 = '' 2023-12-02T22:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-02T22:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 169 = '' 2023-12-02T22:45:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.status 2023-12-02T22:45:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-02T22:45:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=169 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 169 = 169 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-02T22:45:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_26': 2023-12-02T22:45:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127be000a230 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127be000a230 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:17 2023 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:17 2023 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:17 2023 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 315680d7-c2d3-4042-8d33-632d758e85f2 2023-12-02T22:45:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-02T22:45:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:45:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.export 2023-12-02T22:45:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.export 2023-12-02T22:45:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.export 2023-12-02T22:45:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.export 2023-12-02T22:45:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.export 2023-12-02T22:45:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.export 2023-12-02T22:45:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_26.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_26.export 2023-12-02T22:45:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2023-12-02T22:45:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-02T22:45:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-02T22:45:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-02T22:45:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-02T22:45:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_27.status 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_27.status 2023-12-02T22:45:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.status 2023-12-02T22:45:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_27.status 2023-12-02T22:45:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_27.status 2023-12-02T22:45:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-02T22:45:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-02T22:45:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 171 2023-12-02T22:45:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.status 2023-12-02T22:45:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.status 2023-12-02T22:45:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-02T22:45:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=171 2023-12-02T22:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-02T22:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 171 = '' 2023-12-02T22:45:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-02T22:45:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 171 = '' 2023-12-02T22:45:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.status 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-02T22:45:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=171 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 171 = 171 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:45:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-02T22:45:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_27': 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1298dc4b9b4e 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1298dc4b9b4e 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:26 2023 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:26 2023 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:26 2023 2023-12-02T22:45:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:45:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:45:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:45:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: cba1b5db-1e9a-4652-8e1d-956d8a38b3c8 2023-12-02T22:45:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.export 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.export 2023-12-02T22:45:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.export 2023-12-02T22:45:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.export 2023-12-02T22:45:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.export 2023-12-02T22:45:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:45:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.export 2023-12-02T22:45:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_27.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_27.export 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:45:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:45:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-02T22:45:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:45:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-02T22:45:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:45:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:45:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:45:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:45:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:45:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:45:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:45:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:45:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:45:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:45:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:45:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:45:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:45:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:45:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-02T22:45:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:45:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:45:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:45:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:45:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_28.status 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_28.status 2023-12-02T22:45:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.status 2023-12-02T22:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_28.status 2023-12-02T22:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_28.status 2023-12-02T22:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-02T22:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-02T22:45:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 173 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.status 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.status 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-02T22:45:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:45:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=173 2023-12-02T22:45:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:45:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-02T22:45:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=95 2023-12-02T22:45:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 173 = 95 2023-12-02T22:45:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-02T22:45:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=95 2023-12-02T22:45:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 173 = 95 2023-12-02T22:45:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-02T22:45:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:45:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:45:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:45:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:45:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:45:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:45:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:45:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 173 = '' 2023-12-02T22:45:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:45:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.status 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-02T22:46:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=173 2023-12-02T22:46:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 173 = 173 2023-12-02T22:46:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_28': 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a14c4f7def 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a14c4f7def 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:34 2023 2023-12-02T22:46:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:34 2023 2023-12-02T22:46:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:34 2023 2023-12-02T22:46:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 65974775-a20e-4c7a-9d15-5a68ebf338c6 2023-12-02T22:46:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-02T22:46:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.export 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.export 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.export 2023-12-02T22:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.export 2023-12-02T22:46:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.export 2023-12-02T22:46:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.export 2023-12-02T22:46:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_28.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_28.export 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-02T22:46:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-02T22:46:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-02T22:46:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-02T22:46:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-02T22:46:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:46:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:46:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:46:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-02T22:46:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-02T22:46:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-02T22:46:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:46:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:46:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:46:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:46:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:46:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:46:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-02T22:46:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-02T22:46:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:46:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:46:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:46:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:46:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:46:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-12-02T22:46:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_29.status 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_29.status 2023-12-02T22:46:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.status 2023-12-02T22:46:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_29.status 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_29.status 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-02T22:46:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 175 2023-12-02T22:46:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.status 2023-12-02T22:46:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.status 2023-12-02T22:46:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-02T22:46:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=175 2023-12-02T22:46:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-02T22:46:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=96 2023-12-02T22:46:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 175 = 96 2023-12-02T22:46:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-02T22:46:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 175 = '' 2023-12-02T22:46:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-02T22:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 175 = '' 2023-12-02T22:46:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.status 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-02T22:46:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=175 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 175 = 175 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-02T22:46:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_29': 2023-12-02T22:46:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1186a6ec9547 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1186a6ec9547 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:48 2023 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:48 2023 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:48 2023 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8686cb37-c0f3-4cb1-bfb7-7d89f4ef29c2 2023-12-02T22:46:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.export 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.export 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.export 2023-12-02T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.export 2023-12-02T22:46:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.export 2023-12-02T22:46:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.export 2023-12-02T22:46:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_29.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_29.export 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-02T22:46:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-02T22:46:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:46:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:46:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-02T22:46:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:46:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-12-02T22:46:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-02T22:46:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:46:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:46:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:46:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:46:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:46:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:46:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:46:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-12-02T22:46:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-02T22:46:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_30.status 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_30.status 2023-12-02T22:46:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.status 2023-12-02T22:46:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_30.status 2023-12-02T22:46:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_30.status 2023-12-02T22:46:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-02T22:46:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-02T22:46:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 177 2023-12-02T22:46:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.status 2023-12-02T22:46:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.status 2023-12-02T22:46:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-02T22:46:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=177 2023-12-02T22:46:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-02T22:46:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=97 2023-12-02T22:46:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 177 = 97 2023-12-02T22:46:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-02T22:46:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 177 = '' 2023-12-02T22:46:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-02T22:46:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 177 = '' 2023-12-02T22:46:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.status 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-02T22:46:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=177 2023-12-02T22:46:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 177 = 177 2023-12-02T22:46:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_30': 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127ba40c3d5c 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127ba40c3d5c 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:37:56 2023 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:37:56 2023 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:37:56 2023 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: af5e53fb-d079-4e24-8f20-85b325a485b5 2023-12-02T22:46:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.export 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.export 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.export 2023-12-02T22:46:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.export 2023-12-02T22:46:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.export 2023-12-02T22:46:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.export 2023-12-02T22:46:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_30.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_30.export 2023-12-02T22:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2023-12-02T22:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-02T22:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-02T22:46:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-02T22:46:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_31.status 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_31.status 2023-12-02T22:46:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.status 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_31.status 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_31.status 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-02T22:46:22.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 179 2023-12-02T22:46:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.status 2023-12-02T22:46:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.status 2023-12-02T22:46:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-02T22:46:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=179 2023-12-02T22:46:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-02T22:46:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 179 = '' 2023-12-02T22:46:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-02T22:46:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 179 = '' 2023-12-02T22:46:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.status 2023-12-02T22:46:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-02T22:46:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=179 2023-12-02T22:46:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 179 = 179 2023-12-02T22:46:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:25.323 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7008112466606993) > weight (0.25) 2023-12-02T22:46:25.323 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8911011168016763) > weight (0.25) 2023-12-02T22:46:25.324 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T22:46:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T22:46:25.324 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T22:46:25.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:46:25.324+0000 7fa01bc86640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 104468) UID: 1000 2023-12-02T22:46:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-02T22:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_31': 2023-12-02T22:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12981ad21cc1 2023-12-02T22:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12981ad21cc1 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:05 2023 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:05 2023 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:05 2023 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 93fcda49-830e-47ed-9f3a-10f4ff483bab 2023-12-02T22:46:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.export 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.export 2023-12-02T22:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.export 2023-12-02T22:46:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.export 2023-12-02T22:46:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.export 2023-12-02T22:46:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.export 2023-12-02T22:46:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_31.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_31.export 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:26.325 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T22:46:26.326 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:46:26.326 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T22:46:26.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T22:46:26.326 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T22:46:26.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T22:46:26.329 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-02T22:46:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-02T22:46:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-02T22:46:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-02T22:46:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-02T22:46:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_32.status 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_32.status 2023-12-02T22:46:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.status 2023-12-02T22:46:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_32.status 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_32.status 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-02T22:46:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 181 2023-12-02T22:46:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.status 2023-12-02T22:46:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.status 2023-12-02T22:46:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-02T22:46:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=181 2023-12-02T22:46:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-02T22:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 181 = '' 2023-12-02T22:46:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-02T22:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 181 = '' 2023-12-02T22:46:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.status 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-02T22:46:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=181 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 181 = 181 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_32': 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1920ae49c 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1920ae49c 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:11 2023 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:11 2023 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:11 2023 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 662b719c-5386-4ec4-8056-aa4e67a37e59 2023-12-02T22:46:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.export 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.export 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.export 2023-12-02T22:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.export 2023-12-02T22:46:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.export 2023-12-02T22:46:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.export 2023-12-02T22:46:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_32.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_32.export 2023-12-02T22:46:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2023-12-02T22:46:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-02T22:46:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-02T22:46:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-02T22:46:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-02T22:46:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-02T22:46:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-02T22:46:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:46:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-02T22:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_33.status 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_33.status 2023-12-02T22:46:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.status 2023-12-02T22:46:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_33.status 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_33.status 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-02T22:46:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 183 2023-12-02T22:46:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.status 2023-12-02T22:46:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.status 2023-12-02T22:46:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-02T22:46:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=183 2023-12-02T22:46:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-02T22:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 183 = '' 2023-12-02T22:46:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-02T22:46:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 183 = '' 2023-12-02T22:46:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.status 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-02T22:46:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=183 2023-12-02T22:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 183 = 183 2023-12-02T22:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-02T22:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-02T22:46:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_33': 2023-12-02T22:46:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1186ff1bf91a 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1186ff1bf91a 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:24 2023 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:24 2023 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:24 2023 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d64bed83-0a8b-46ea-b180-1169dba0dac7 2023-12-02T22:46:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.export 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.export 2023-12-02T22:46:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.export 2023-12-02T22:46:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.export 2023-12-02T22:46:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.export 2023-12-02T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.export 2023-12-02T22:46:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_33.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_33.export 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-02T22:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-02T22:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-02T22:46:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-02T22:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:46:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:46:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:46:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:46:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:46:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:46:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:46:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-02T22:46:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-02T22:46:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:46:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:46:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:46:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:46:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:46:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:46:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-02T22:46:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_34.status 2023-12-02T22:46:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_34.status 2023-12-02T22:46:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.status 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_34.status 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_34.status 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-02T22:46:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 185 2023-12-02T22:46:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.status 2023-12-02T22:46:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.status 2023-12-02T22:46:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-02T22:46:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=185 2023-12-02T22:46:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-02T22:46:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 185 = '' 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-02T22:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 185 = '' 2023-12-02T22:46:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.status 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-02T22:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=185 2023-12-02T22:46:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 185 = 185 2023-12-02T22:46:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-02T22:46:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_34': 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 127bd9df546c 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.127bd9df546c 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:33 2023 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:33 2023 2023-12-02T22:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:33 2023 2023-12-02T22:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 72af8c9e-08e6-4b8b-a6d4-3048fd53a1e1 2023-12-02T22:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.export 2023-12-02T22:46:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.export 2023-12-02T22:46:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.export 2023-12-02T22:46:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.export 2023-12-02T22:46:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.export 2023-12-02T22:46:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.export 2023-12-02T22:46:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_34.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_34.export 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-02T22:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-02T22:46:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-02T22:46:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_35.status 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_35.status 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.status 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_35.status 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_35.status 2023-12-02T22:46:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-02T22:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-02T22:46:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 187 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.status 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.status 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-02T22:46:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=187 2023-12-02T22:46:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-02T22:46:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 187 = '' 2023-12-02T22:46:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-02T22:46:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 187 = '' 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.status 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-02T22:46:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=187 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 187 = 187 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:46:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-02T22:46:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_35': 2023-12-02T22:46:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:46:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12982586b1d1 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12982586b1d1 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:40 2023 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:40 2023 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:40 2023 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4bb9743c-64d8-40e8-b02a-6a91310c2444 2023-12-02T22:46:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:46:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.export 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.export 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.export 2023-12-02T22:46:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.export 2023-12-02T22:46:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.export 2023-12-02T22:46:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:46:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.export 2023-12-02T22:46:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_35.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_35.export 2023-12-02T22:46:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:46:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:46:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:46:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-02T22:46:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:46:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:46:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:46:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:46:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:46:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:46:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:46:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:46:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-02T22:46:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:46:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:46:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:46:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:46:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_36.status 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_36.status 2023-12-02T22:46:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.status 2023-12-02T22:46:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_36.status 2023-12-02T22:46:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_36.status 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-02T22:46:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 189 2023-12-02T22:46:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.status 2023-12-02T22:46:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:46:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.status 2023-12-02T22:46:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-02T22:46:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:46:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=189 2023-12-02T22:46:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:46:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-02T22:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=103 2023-12-02T22:46:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 189 = 103 2023-12-02T22:46:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:46:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-02T22:46:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=103 2023-12-02T22:46:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 189 = 103 2023-12-02T22:46:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-02T22:46:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:46:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:46:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:46:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:46:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:46:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:46:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:46:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 189 = '' 2023-12-02T22:46:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:46:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.status 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-02T22:47:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=189 2023-12-02T22:47:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 189 = 189 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:47:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-02T22:47:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_36': 2023-12-02T22:47:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:47:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:47:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:47:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1fb6bf7f9 2023-12-02T22:47:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1fb6bf7f9 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:50 2023 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:50 2023 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:50 2023 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4b378cd6-dd34-4f37-b8aa-ec8c5a2768b0 2023-12-02T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:47:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.export 2023-12-02T22:47:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.export 2023-12-02T22:47:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.export 2023-12-02T22:47:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.export 2023-12-02T22:47:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.export 2023-12-02T22:47:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.export 2023-12-02T22:47:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_36.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_36.export 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:47:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-02T22:47:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-02T22:47:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:47:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:04.220 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2542280132751902) > weight (0.25) 2023-12-02T22:47:04.220 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4146378480706665) > weight (0.25) 2023-12-02T22:47:04.220 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T22:47:04.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T22:47:04.220 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:47:04.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:47:04.220+0000 7f574b25c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 99408) UID: 1000 2023-12-02T22:47:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:47:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:47:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:47:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:47:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:47:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-02T22:47:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-02T22:47:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-02T22:47:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-02T22:47:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-02T22:47:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:47:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:47:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:47:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-12-02T22:47:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-02T22:47:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2023-12-02T22:47:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:47:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:47:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:47:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2023-12-02T22:47:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:47:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:47:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:47:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:47:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:47:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:47:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:47:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_37.status 2023-12-02T22:47:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_37.status 2023-12-02T22:47:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.status 2023-12-02T22:47:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_37.status 2023-12-02T22:47:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_37.status 2023-12-02T22:47:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-02T22:47:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-02T22:47:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 191 2023-12-02T22:47:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.status 2023-12-02T22:47:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.status 2023-12-02T22:47:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-02T22:47:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:47:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=191 2023-12-02T22:47:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:47:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-02T22:47:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=104 2023-12-02T22:47:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 191 = 104 2023-12-02T22:47:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:47:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-02T22:47:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 191 = '' 2023-12-02T22:47:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-02T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 191 = '' 2023-12-02T22:47:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.status 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-02T22:47:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=191 2023-12-02T22:47:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 191 = 191 2023-12-02T22:47:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-02T22:47:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:47:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-02T22:47:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_37': 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 11862fe86e5b 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.11862fe86e5b 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:38:59 2023 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:38:59 2023 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:38:59 2023 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:47:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 6de1f79c-583e-4974-ae84-18b971385b78 2023-12-02T22:47:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:47:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-02T22:47:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:47:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.export 2023-12-02T22:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.export 2023-12-02T22:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.export 2023-12-02T22:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.export 2023-12-02T22:47:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.export 2023-12-02T22:47:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.export 2023-12-02T22:47:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_37.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_37.export 2023-12-02T22:47:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:47:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:47:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:47:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-02T22:47:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:47:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:47:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:47:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:47:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:47:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:47:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-02T22:47:13.221 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T22:47:13.222 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:47:13.222 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-02T22:47:13.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T22:47:13.222 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T22:47:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T22:47:13.225 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-12-02T22:47:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-02T22:47:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-12-02T22:47:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:47:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:47:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:47:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:47:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:47:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:47:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:47:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-02T22:47:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_38.status 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_38.status 2023-12-02T22:47:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.status 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_38.status 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_38.status 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-02T22:47:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 193 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.status 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.status 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-02T22:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:47:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=193 2023-12-02T22:47:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-02T22:47:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=105 2023-12-02T22:47:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 193 = 105 2023-12-02T22:47:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:47:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-02T22:47:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 193 = '' 2023-12-02T22:47:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-02T22:47:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 193 = '' 2023-12-02T22:47:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.status 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-02T22:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=193 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 193 = 193 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:47:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-02T22:47:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_38': 2023-12-02T22:47:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12bab3efd2e1 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12bab3efd2e1 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:35 2023 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:35 2023 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:35 2023 2023-12-02T22:47:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:47:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:47:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:47:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dedd0af-fa04-42db-9022-5c3e623cc7a9 2023-12-02T22:47:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:47:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-02T22:47:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:47:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:47:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.export 2023-12-02T22:47:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.export 2023-12-02T22:47:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.export 2023-12-02T22:47:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.export 2023-12-02T22:47:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.export 2023-12-02T22:47:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.export 2023-12-02T22:47:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_38.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_38.export 2023-12-02T22:47:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:47:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2023-12-02T22:47:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-02T22:47:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:47:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-02T22:47:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-02T22:47:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:47:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:47:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:47:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-02T22:47:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2023-12-02T22:47:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:47:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:47:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:47:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:47:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2023-12-02T22:47:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 1 2023-12-02T22:47:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-02T22:47:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-02T22:47:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-02T22:47:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:47:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:47:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:47:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:47:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:47:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:47:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:47:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:47:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-02T22:47:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:47:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:47:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:47:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2023-12-02T22:47:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 2 2023-12-02T22:47:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-02T22:47:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-02T22:47:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-02T22:47:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-02T22:47:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-02T22:47:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:47:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:47:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:47:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:47:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-02T22:47:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2023-12-02T22:47:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:47:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:47:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2023-12-02T22:47:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 4 2023-12-02T22:47:27.330 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7803321891829113) > weight (0.25) 2023-12-02T22:47:27.330 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3124972602244819) > weight (0.25) 2023-12-02T22:47:27.330 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.285928083423155) > weight (0.25) 2023-12-02T22:47:27.330 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6623820859788285) > weight (0.25) 2023-12-02T22:47:27.330 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-02T22:47:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-02T22:47:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-02T22:47:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-02T22:47:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:47:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:47:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:47:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:47:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-02T22:47:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2023-12-02T22:47:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:47:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:47:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:47:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2023-12-02T22:47:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 8 2023-12-02T22:47:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-02T22:47:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-02T22:47:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-02T22:47:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:47:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:47:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:47:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:47:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-02T22:47:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-02T22:47:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2023-12-02T22:47:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:47:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:47:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:47:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:47:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:47:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:47:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2023-12-02T22:47:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:47:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:47:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:47:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:47:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_39.status 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_39.status 2023-12-02T22:47:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.status 2023-12-02T22:47:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_39.status 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_39.status 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-02T22:47:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 195 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.status 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.status 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-02T22:47:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:47:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=195 2023-12-02T22:47:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-02T22:47:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 195 = '' 2023-12-02T22:47:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:47:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-02T22:47:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 195 = '' 2023-12-02T22:47:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.status 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-02T22:47:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=195 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 195 = 195 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:47:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_39': 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12babc6401ee 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12babc6401ee 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:47:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:18 2023 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:18 2023 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:18 2023 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e5ea4907-8178-4427-a782-ee8006b75e82 2023-12-02T22:47:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.export 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.export 2023-12-02T22:47:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.export 2023-12-02T22:47:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.export 2023-12-02T22:47:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.export 2023-12-02T22:47:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.export 2023-12-02T22:47:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_39.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_39.export 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:47:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-02T22:47:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-02T22:47:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-12-02T22:47:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:47:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:47:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:47:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:47:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:47:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:47:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-02T22:47:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-02T22:47:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-12-02T22:47:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-12-02T22:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:47:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:47:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:47:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_40.status 2023-12-02T22:47:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_40.status 2023-12-02T22:47:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.status 2023-12-02T22:47:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_40.status 2023-12-02T22:47:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_40.status 2023-12-02T22:47:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-02T22:47:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-02T22:47:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 197 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.status 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.status 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-02T22:47:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:47:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=197 2023-12-02T22:47:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:47:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-02T22:47:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 197 = '' 2023-12-02T22:47:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-02T22:47:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 197 = '' 2023-12-02T22:47:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:47:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.status 2023-12-02T22:47:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-02T22:47:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=197 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 197 = 197 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:47:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_40': 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12ba26b87102 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12ba26b87102 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:32 2023 2023-12-02T22:47:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:32 2023 2023-12-02T22:47:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:32 2023 2023-12-02T22:47:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:47:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:47:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:47:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7f8c70ee-6301-46d2-8c34-89c793426067 2023-12-02T22:47:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.export 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.export 2023-12-02T22:47:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.export 2023-12-02T22:47:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.export 2023-12-02T22:47:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.export 2023-12-02T22:47:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.export 2023-12-02T22:47:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_40.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_40.export 2023-12-02T22:47:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:47:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2023-12-02T22:47:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-02T22:47:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:47:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:47:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:47:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-02T22:47:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-02T22:47:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-12-02T22:47:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-12-02T22:47:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:47:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:47:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:47:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:47:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:47:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:47:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:47:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-02T22:47:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_41.status 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_41.status 2023-12-02T22:47:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.status 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_41.status 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_41.status 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-02T22:47:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 199 2023-12-02T22:47:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.status 2023-12-02T22:47:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.status 2023-12-02T22:47:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-02T22:47:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:47:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=199 2023-12-02T22:47:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-02T22:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 199 = '' 2023-12-02T22:47:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:47:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-02T22:47:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:47:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:47:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:47:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:47:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 199 = '' 2023-12-02T22:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.status 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-02T22:47:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:47:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:47:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=199 2023-12-02T22:47:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 199 = 199 2023-12-02T22:47:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-02T22:47:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:47:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-02T22:47:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_41': 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1186250b132c 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1186250b132c 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:41 2023 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:41 2023 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:41 2023 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:47:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:47:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a49a8b18-32ec-4784-9e10-57e3c1d4cf26 2023-12-02T22:47:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:47:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-02T22:47:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.export 2023-12-02T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.export 2023-12-02T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.export 2023-12-02T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.export 2023-12-02T22:47:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.export 2023-12-02T22:47:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:47:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.export 2023-12-02T22:47:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_41.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_41.export 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:47:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-02T22:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-02T22:47:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:47:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:47:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:47:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:47:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-02T22:47:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:47:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:47:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:47:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:47:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-02T22:47:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:47:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:47:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:47:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:47:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-02T22:47:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-02T22:47:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:47:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:47:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:47:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:47:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:47:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:47:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:47:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:47:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-12-02T22:48:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_42.status 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_42.status 2023-12-02T22:48:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.status 2023-12-02T22:48:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_42.status 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_42.status 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-02T22:48:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 201 2023-12-02T22:48:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.status 2023-12-02T22:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.status 2023-12-02T22:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-02T22:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=201 2023-12-02T22:48:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-02T22:48:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 201 = '' 2023-12-02T22:48:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-02T22:48:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 201 = '' 2023-12-02T22:48:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.status 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-02T22:48:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=201 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 201 = 201 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-02T22:48:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_42': 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1298bff6d676 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1298bff6d676 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:39:51 2023 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:39:51 2023 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:39:51 2023 2023-12-02T22:48:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7d8c8ff7-9e39-4fd4-a378-75b3505c1c8c 2023-12-02T22:48:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-02T22:48:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:48:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.export 2023-12-02T22:48:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.export 2023-12-02T22:48:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.export 2023-12-02T22:48:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.export 2023-12-02T22:48:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.export 2023-12-02T22:48:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.export 2023-12-02T22:48:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_42.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_42.export 2023-12-02T22:48:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2023-12-02T22:48:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-02T22:48:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-02T22:48:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-02T22:48:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-02T22:48:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-02T22:48:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-02T22:48:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_43.status 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_43.status 2023-12-02T22:48:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.status 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_43.status 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_43.status 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-02T22:48:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 203 2023-12-02T22:48:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.status 2023-12-02T22:48:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.status 2023-12-02T22:48:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-02T22:48:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=203 2023-12-02T22:48:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-02T22:48:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 203 = '' 2023-12-02T22:48:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-02T22:48:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 203 = '' 2023-12-02T22:48:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.status 2023-12-02T22:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-02T22:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=203 2023-12-02T22:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 203 = 203 2023-12-02T22:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-02T22:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-02T22:48:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_43': 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1cde1cf7e 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1cde1cf7e 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:01 2023 2023-12-02T22:48:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:01 2023 2023-12-02T22:48:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:01 2023 2023-12-02T22:48:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: dabad757-1fd3-41ef-b3ef-13b1187bb4f2 2023-12-02T22:48:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-02T22:48:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.export 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.export 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.export 2023-12-02T22:48:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.export 2023-12-02T22:48:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.export 2023-12-02T22:48:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.export 2023-12-02T22:48:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_43.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_43.export 2023-12-02T22:48:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2023-12-02T22:48:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-02T22:48:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-02T22:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-02T22:48:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-02T22:48:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-02T22:48:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-02T22:48:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-02T22:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-02T22:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-02T22:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-02T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_44.status 2023-12-02T22:48:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_44.status 2023-12-02T22:48:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.status 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_44.status 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_44.status 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-02T22:48:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-02T22:48:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 205 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.status 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.status 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-12-02T22:48:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=205 2023-12-02T22:48:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-02T22:48:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=111 2023-12-02T22:48:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 205 = 111 2023-12-02T22:48:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-02T22:48:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 205 = '' 2023-12-02T22:48:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-02T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 205 = '' 2023-12-02T22:48:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.status 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-02T22:48:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=205 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 205 = 205 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_44': 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12bad60cfbf1 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12bad60cfbf1 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:09 2023 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:09 2023 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:09 2023 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: a442254e-d4a3-4fcf-9c21-028b8a99b2f0 2023-12-02T22:48:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.export 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.export 2023-12-02T22:48:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.export 2023-12-02T22:48:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.export 2023-12-02T22:48:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.export 2023-12-02T22:48:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.export 2023-12-02T22:48:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_44.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_44.export 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-02T22:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-02T22:48:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-02T22:48:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-12-02T22:48:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-02T22:48:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-12-02T22:48:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-02T22:48:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_45.status 2023-12-02T22:48:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_45.status 2023-12-02T22:48:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.status 2023-12-02T22:48:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_45.status 2023-12-02T22:48:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_45.status 2023-12-02T22:48:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-02T22:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-02T22:48:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 207 2023-12-02T22:48:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.status 2023-12-02T22:48:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.status 2023-12-02T22:48:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-02T22:48:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=207 2023-12-02T22:48:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-02T22:48:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 207 = '' 2023-12-02T22:48:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-02T22:48:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 207 = '' 2023-12-02T22:48:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.status 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-02T22:48:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=207 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 207 = 207 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_45': 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12c213c6c6a5 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12c213c6c6a5 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:49 2023 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:49 2023 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:49 2023 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 09f87dc1-458a-4e3a-a6c6-7afee7b3129e 2023-12-02T22:48:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.export 2023-12-02T22:48:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.export 2023-12-02T22:48:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.export 2023-12-02T22:48:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.export 2023-12-02T22:48:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.export 2023-12-02T22:48:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.export 2023-12-02T22:48:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_45.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_45.export 2023-12-02T22:48:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-02T22:48:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-02T22:48:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-02T22:48:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-12-02T22:48:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-02T22:48:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2023-12-02T22:48:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2023-12-02T22:48:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-02T22:48:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:48:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:48:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:48:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2023-12-02T22:48:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-02T22:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_46.status 2023-12-02T22:48:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_46.status 2023-12-02T22:48:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.status 2023-12-02T22:48:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_46.status 2023-12-02T22:48:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_46.status 2023-12-02T22:48:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-02T22:48:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-02T22:48:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 209 2023-12-02T22:48:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.status 2023-12-02T22:48:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.status 2023-12-02T22:48:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-02T22:48:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=209 2023-12-02T22:48:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-02T22:48:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 209 = '' 2023-12-02T22:48:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-02T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 209 = '' 2023-12-02T22:48:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.status 2023-12-02T22:48:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-02T22:48:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=209 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 209 = 209 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-02T22:48:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_46': 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12c2cffe078a 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12c2cffe078a 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:25 2023 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:25 2023 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:25 2023 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c1d2a65d-c465-4d9a-8cce-99526d99432c 2023-12-02T22:48:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-02T22:48:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.export 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.export 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.export 2023-12-02T22:48:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.export 2023-12-02T22:48:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.export 2023-12-02T22:48:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.export 2023-12-02T22:48:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_46.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_46.export 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-02T22:48:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-02T22:48:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-02T22:48:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-12-02T22:48:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-02T22:48:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-02T22:48:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-02T22:48:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-02T22:48:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-02T22:48:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-02T22:48:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-02T22:48:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:48:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:48:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:48:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2023-12-02T22:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-02T22:48:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_47.status 2023-12-02T22:48:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_47.status 2023-12-02T22:48:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.status 2023-12-02T22:48:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_47.status 2023-12-02T22:48:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_47.status 2023-12-02T22:48:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-02T22:48:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-02T22:48:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 211 2023-12-02T22:48:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.status 2023-12-02T22:48:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.status 2023-12-02T22:48:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-02T22:48:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=211 2023-12-02T22:48:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-02T22:48:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 211 = '' 2023-12-02T22:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-02T22:48:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 211 = '' 2023-12-02T22:48:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.status 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-02T22:48:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=211 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 211 = 211 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:39.225 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T22:48:39.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T22:48:39.226 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T22:48:39.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:48:39.226+0000 7fa796032640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 108097) UID: 1000 2023-12-02T22:48:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-02T22:48:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_47': 2023-12-02T22:48:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12c2dbcef33b 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12c2dbcef33b 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:37 2023 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:37 2023 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:37 2023 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 9431194f-1321-4cad-a347-45f6490d98bd 2023-12-02T22:48:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.export 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.export 2023-12-02T22:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.export 2023-12-02T22:48:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.export 2023-12-02T22:48:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.export 2023-12-02T22:48:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.export 2023-12-02T22:48:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_47.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_47.export 2023-12-02T22:48:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-02T22:48:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-02T22:48:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-02T22:48:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-02T22:48:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2023-12-02T22:48:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-02T22:48:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2023-12-02T22:48:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-02T22:48:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:48:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:48:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2023-12-02T22:48:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-02T22:48:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_48.status 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_48.status 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.status 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_48.status 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_48.status 2023-12-02T22:48:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-02T22:48:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-02T22:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 213 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.status 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.status 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-02T22:48:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:44.227 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T22:48:44.227 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:48:44.227 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T22:48:44.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T22:48:44.228 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T22:48:44.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T22:48:44.230 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-02T22:48:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=213 2023-12-02T22:48:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-02T22:48:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=115 2023-12-02T22:48:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 213 = 115 2023-12-02T22:48:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-02T22:48:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=115 2023-12-02T22:48:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 213 = 115 2023-12-02T22:48:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-02T22:48:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 213 = '' 2023-12-02T22:48:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1.6 2023-12-02T22:48:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.status 2023-12-02T22:48:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-02T22:48:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=213 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 213 = 213 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-02T22:48:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_48': 2023-12-02T22:48:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12c21f027a6a 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12c21f027a6a 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:47 2023 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:47 2023 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:47 2023 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e848264b-794b-4557-91db-d8f1e8b231f2 2023-12-02T22:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.export 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.export 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.export 2023-12-02T22:48:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.export 2023-12-02T22:48:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.export 2023-12-02T22:48:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.export 2023-12-02T22:48:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_48.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_48.export 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-02T22:48:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-02T22:48:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-02T22:48:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-02T22:48:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-02T22:48:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-12-02T22:48:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-02T22:48:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2023-12-02T22:48:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-02T22:48:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2023-12-02T22:48:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:3 2023-12-02T22:48:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:3 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=3 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=3 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:3 2023-12-02T22:48:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:3 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=3 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=3 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=3 2023-12-02T22:48:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T22:48:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T22:48:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2023-12-02T22:48:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-02T22:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_49.status 2023-12-02T22:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_49.status 2023-12-02T22:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.status 2023-12-02T22:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_49.status 2023-12-02T22:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_49.status 2023-12-02T22:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-02T22:48:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-02T22:48:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 215 2023-12-02T22:48:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.status 2023-12-02T22:48:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.status 2023-12-02T22:48:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-02T22:48:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=215 2023-12-02T22:48:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:48:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-02T22:48:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 215 = '' 2023-12-02T22:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-02T22:48:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:48:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:48:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:48:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:48:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 215 = '' 2023-12-02T22:48:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:48:53.883 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:48:53.881+0000 7f2e21bfd640 -1 reset not still connected to 0x557d1a83d110 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.status 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-02T22:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:48:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=215 2023-12-02T22:48:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 215 = 215 2023-12-02T22:48:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-02T22:48:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:48:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-02T22:48:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_49': 2023-12-02T22:48:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:48:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:48:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1298df142c43 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1298df142c43 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:40:54 2023 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:40:54 2023 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:40:54 2023 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8c19922c-ba85-47c0-866f-53244088c863 2023-12-02T22:48:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.export 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.export 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.export 2023-12-02T22:48:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.export 2023-12-02T22:48:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.export 2023-12-02T22:48:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:48:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.export 2023-12-02T22:48:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_49.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_49.export 2023-12-02T22:48:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:48:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2023-12-02T22:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-02T22:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T22:48:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-02T22:48:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2023-12-02T22:48:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2023-12-02T22:48:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T22:48:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2023-12-02T22:48:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T22:48:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 3 2023-12-02T22:48:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2023-12-02T22:48:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T22:48:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2023-12-02T22:48:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2023-12-02T22:48:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T22:48:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T22:48:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-02T22:48:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-02T22:48:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T22:48:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2023-12-02T22:48:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T22:48:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T22:48:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-12-02T22:48:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-02T22:48:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T22:48:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:2 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:2 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2023-12-02T22:48:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=2 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=2 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:2 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:2 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2023-12-02T22:48:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=2 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=2 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=2 2023-12-02T22:48:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T22:48:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T22:48:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2023-12-02T22:48:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2023-12-02T22:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T22:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2023-12-02T22:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2023-12-02T22:48:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2023-12-02T22:48:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2023-12-02T22:48:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = journal ']' 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster2-mirror-image_50.status 2023-12-02T22:48:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster2-mirror-image_50.status 2023-12-02T22:48:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.status 2023-12-02T22:48:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mkfname cluster1-mirror-image_50.status 2023-12-02T22:48:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo cluster1-mirror-image_50.status 2023-12-02T22:48:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T22:48:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_status_log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:48:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-02T22:48:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-02T22:48:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Snapshot ID: 217 2023-12-02T22:48:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.status 2023-12-02T22:48:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:48:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:48:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:48:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.status 2023-12-02T22:48:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-02T22:48:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:48:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T22:48:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snapshot_id=217 2023-12-02T22:48:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:48:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:48:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2023-12-02T22:49:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:49:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:49:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:49:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:49:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-02T22:49:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:49:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:49:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:49:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:49:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:49:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:49:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:49:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 217 = '' 2023-12-02T22:49:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:49:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2023-12-02T22:49:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:49:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:49:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:49:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:49:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-02T22:49:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:49:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:49:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:49:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:-:1.1: Document is empty 2023-12-02T22:49:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2023-12-02T22:49:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:^ 2023-12-02T22:49:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id= 2023-12-02T22:49:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 217 = '' 2023-12-02T22:49:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T22:49:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.8 2023-12-02T22:49:00.859 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb28f7dee0 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da60dd0 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da60ea0 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da612b0 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da79c70 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da8e410 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da8ec30 2023-12-02T22:49:00.860 INFO:tasks.ceph.cluster1.osd.0.smithi012.stderr:2023-12-02T22:49:00.858+0000 7f1ed3491640 -1 reset not still connected to 0x55cb2da8fa00 2023-12-02T22:49:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:49:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:49:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:49:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local log=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.status 2023-12-02T22:49:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-02T22:49:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T22:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T22:49:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local primary_snapshot_id=217 2023-12-02T22:49:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 217 = 217 2023-12-02T22:49:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-02T22:49:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:49:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:49:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:49:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:49:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T22:49:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:49:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_50': 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 2 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 12a1fee92756 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.12a1fee92756 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:49:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:41:01 2023 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:41:01 2023 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:41:01 2023 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4b794018-5af4-4f87-a8d7-86141ee6bc32 2023-12-02T22:49:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.export 2023-12-02T22:49:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.export 2023-12-02T22:49:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.export 2023-12-02T22:49:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.export 2023-12-02T22:49:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:49:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.export 2023-12-02T22:49:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T22:49:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.export 2023-12-02T22:49:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.cQPVpMBksv/cluster2-mirror-image_50.export /tmp/tmp.cQPVpMBksv/cluster1-mirror-image_50.export 2023-12-02T22:49:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: delete many images' 2023-12-02T22:49:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:49:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:49:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:49:02 TEST: delete many images 2023-12-02T22:49:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:49:02 TEST: delete many images 2023-12-02T22:49:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2023-12-02T22:49:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2023-12-02T22:49:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-02T22:49:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:49:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-02T22:49:03.331 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5261911647208268) > weight (0.25) 2023-12-02T22:49:03.331 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T22:49:03.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T22:49:03.332 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T22:49:03.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:49:03.331+0000 7f02b78a0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 95690) UID: 1000 2023-12-02T22:49:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-02T22:49:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:49:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_1 2023-12-02T22:49:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:49:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-02T22:49:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-02T22:49:03.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:49:03.924+0000 7fbc5bb2a640 -1 rbd::mirror::ImageMap: 0x55e61e322240 handle_peer_ack_remove: failed to remove global_image_id=1b41b123-8764-4c2a-808d-b6f2a05a0f17 2023-12-02T22:49:04.332 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T22:49:04.333 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:49:04.333 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T22:49:04.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T22:49:04.333 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T22:49:04.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T22:49:04.336 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-02T22:49:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-02T22:49:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-02T22:49:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:49:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_2 2023-12-02T22:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-02T22:49:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-02T22:49:06.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:06.988+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324aab89c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:06.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:06.988+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a698500 [3/f793e7a7-0001-44a2-82ae-444e4e65879f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-02T22:49:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-02T22:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:49:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_3 2023-12-02T22:49:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:49:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-02T22:49:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-02T22:49:09.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:09.985+0000 7f20dfb2c640 -1 rbd::mirror::ImageMap: 0x55ff14facea0 handle_peer_ack_remove: failed to remove global_image_id=c028f4b4-ce10-4094-9ec7-655b7f816feb 2023-12-02T22:49:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2023-12-02T22:49:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-02T22:49:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:49:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-02T22:49:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-02T22:49:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:49:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_4 2023-12-02T22:49:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:49:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-02T22:49:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-02T22:49:12.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:12.359+0000 7f20dfb2c640 -1 rbd::mirror::ImageMap: 0x55ff14facea0 handle_peer_ack_remove: failed to remove global_image_id=a8f37621-7a4e-4f6c-b01d-f20a1d76f088 2023-12-02T22:49:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2023-12-02T22:49:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-02T22:49:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:49:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-02T22:49:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-02T22:49:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:49:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_5 2023-12-02T22:49:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:49:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-02T22:49:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-02T22:49:15.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:15.363+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff14cd6820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:15.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:15.363+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff15542500 [3/3083f507-4484-4113-938a-b4257c974175] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-02T22:49:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-02T22:49:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:49:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_6 2023-12-02T22:49:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:49:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-02T22:49:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-02T22:49:18.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:18.373+0000 7f20e032d640 -1 rbd::mirror::ImageMap: 0x55ff14facea0 handle_peer_ack_remove: failed to remove global_image_id=6f4a39a9-03a2-4b4f-a1b0-8e15b573f1ab 2023-12-02T22:49:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2023-12-02T22:49:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-02T22:49:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:49:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-02T22:49:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-02T22:49:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_7 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:49:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-02T22:49:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-02T22:49:21.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:21.392+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40394820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:21.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:21.392+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40435900 [3/29d0f358-af7a-4fe0-84a4-8618fbdbda52] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:49:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-02T22:49:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-02T22:49:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:49:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_8 2023-12-02T22:49:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:49:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-02T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-02T22:49:24.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:24.395+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee4036b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:24.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:24.395+0000 7faf26ce2640 -1 rbd::mirror::ImageReplayer: 0x55ee40434780 [3/54d6e2fd-a8c4-41f2-93f4-8602f730be4d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-02T22:49:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-02T22:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_9 2023-12-02T22:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-02T22:49:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-02T22:49:27.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:27.408+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab22b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:27.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:27.408+0000 7f6bdd535640 -1 rbd::mirror::ImageReplayer: 0x56324a54a780 [3/f162a3b0-3c88-4e6c-ae84-ffb10c73c491] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-02T22:49:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-02T22:49:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:49:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_10 2023-12-02T22:49:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:49:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-02T22:49:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-02T22:49:30.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:30.365+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab22d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:30.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:30.365+0000 7f6bdd535640 -1 rbd::mirror::ImageReplayer: 0x56324a699400 [3/744cfe14-46e0-4471-aa62-cd6ed74706ca] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-02T22:49:32.470 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:49:32.468+0000 7eff3a87e640 -1 reset not still connected to 0x559712beeb60 2023-12-02T22:49:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-02T22:49:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:49:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_11 2023-12-02T22:49:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:49:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-02T22:49:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-02T22:49:33.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:33.418+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff158531e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:33.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:33.418+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff15542c80 [3/7d477ce5-b8e7-48b3-aa17-29ac3ee8f70c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:49:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-02T22:49:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-02T22:49:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:49:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_12 2023-12-02T22:49:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:49:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-02T22:49:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-02T22:49:36.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:49:36.427+0000 7f8734d3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c456d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:36.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:49:36.427+0000 7f873bd49640 -1 rbd::mirror::ImageReplayer: 0x55f34bae0c80 [3/87b8efc0-f045-4058-89b2-d6580e15413a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-02T22:49:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-02T22:49:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_13 2023-12-02T22:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:49:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-02T22:49:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-02T22:49:39.405 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:49:39.404+0000 7f315ad82640 -1 reset not still connected to 0x564dfd805e10 2023-12-02T22:49:39.406 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:49:39.404+0000 7f315ad82640 -1 reset not still connected to 0x564e0f159040 2023-12-02T22:49:39.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:39.444+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff15853380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:39.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:39.445+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff1531e280 [3/d76b9c8b-eb85-4586-95f4-981ca6150274] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-02T22:49:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-02T22:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:49:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_14 2023-12-02T22:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-02T22:49:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-02T22:49:42.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:42.448+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab229c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:42.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:42.448+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54af00 [3/da7e5eab-bc9e-4019-826f-9d3346e2d46b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-02T22:49:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-02T22:49:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:49:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_15 2023-12-02T22:49:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:49:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-02T22:49:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-02T22:49:45.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:45.367+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab224e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:45.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:49:45.367+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54b680 [3/b7fa7b77-c7c1-4f78-b242-68962c8e8eb8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2023-12-02T22:49:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-02T22:49:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:49:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-02T22:49:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-02T22:49:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:49:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_16 2023-12-02T22:49:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:49:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-02T22:49:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-02T22:49:48.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:48.452+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee403949c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:48.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:48.452+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40274a00 [3/99280131-88b8-4f83-930d-440a9d1456ee] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-02T22:49:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-02T22:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_17 2023-12-02T22:49:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:49:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-02T22:49:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-02T22:49:51.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:51.673+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff15852b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:51.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:49:51.673+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff15543400 [3/06d1cdd3-95e7-4459-aa25-f7cf37c99105] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-02T22:49:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-02T22:49:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:49:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_18 2023-12-02T22:49:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:49:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-02T22:49:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-02T22:49:54.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:54.477+0000 7faf204d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee405d29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:54.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:54.477+0000 7faf26ce2640 -1 rbd::mirror::ImageReplayer: 0x55ee40435900 [3/ebdd1cd5-86de-4c0d-bc2d-c273a9a23947] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:49:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-02T22:49:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-02T22:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:49:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_19 2023-12-02T22:49:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:49:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-02T22:49:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-02T22:49:57.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:57.365+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40394b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:49:57.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:49:57.365+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40435180 [3/a8c6040f-b031-4199-aca7-736fca431324] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:49:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:49:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-02T22:50:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-02T22:50:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:50:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_20 2023-12-02T22:50:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:50:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-02T22:50:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-02T22:50:00.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:50:00.369+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324a9a71e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:00.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:50:00.369+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a698f00 [3/89f1b9c1-adde-4973-967a-efbd5a93afd6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-02T22:50:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-02T22:50:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:50:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_21 2023-12-02T22:50:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:50:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-02T22:50:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-02T22:50:03.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:03.393+0000 7f873553c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c456ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:03.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:03.393+0000 7f873bd49640 -1 rbd::mirror::ImageReplayer: 0x55f34bae0780 [3/758205de-a838-479c-8d08-b70c14a4aacc] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:50:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-02T22:50:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-02T22:50:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_22 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:50:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-02T22:50:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-02T22:50:06.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:06.408+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40394d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:06.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:06.408+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40435b80 [3/5661f1b2-001e-4d14-9686-8b906099edb6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-02T22:50:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-02T22:50:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:50:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_23 2023-12-02T22:50:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:50:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-02T22:50:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-02T22:50:09.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6810435768841598) > weight (0.25) 2023-12-02T22:50:09.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5861181774942211) > weight (0.25) 2023-12-02T22:50:09.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3165306211632185) > weight (0.25) 2023-12-02T22:50:09.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5725118727731683) > weight (0.25) 2023-12-02T22:50:09.337 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-12-02T22:50:09.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:09.406+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40211a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:09.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:09.406+0000 7faf26ce2640 -1 rbd::mirror::ImageReplayer: 0x55ee40434a00 [3/645a0ff2-7b25-4678-9148-2db63c48f624] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:11.230 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.41552722517991936) > weight (0.25) 2023-12-02T22:50:11.231 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.834388306350006) > weight (0.25) 2023-12-02T22:50:11.231 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T22:50:11.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T22:50:11.231 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:50:11.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:50:11.231+0000 7f5c90271640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 113559) UID: 1000 2023-12-02T22:50:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2023-12-02T22:50:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-02T22:50:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:50:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-02T22:50:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-02T22:50:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:50:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_24 2023-12-02T22:50:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:50:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-02T22:50:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-02T22:50:12.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:12.413+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff158b11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:12.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:12.413+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff15543180 [3/208a8244-37c7-4cb5-8dbd-872c1e9d06f2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2023-12-02T22:50:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-02T22:50:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:50:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-02T22:50:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-02T22:50:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:50:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_25 2023-12-02T22:50:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:50:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-02T22:50:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-02T22:50:15.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:50:15.318+0000 7fbc5bb2a640 -1 rbd::mirror::ImageMap: 0x55e61e322240 handle_peer_ack_remove: failed to remove global_image_id=c16dc03c-c292-424e-be08-d6e908fda7e8 2023-12-02T22:50:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-02T22:50:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-02T22:50:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:50:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_26 2023-12-02T22:50:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:50:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-02T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-02T22:50:18.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:50:18.385+0000 7fbc5bb2a640 -1 rbd::mirror::ImageMap: 0x55e61e322240 handle_peer_ack_remove: failed to remove global_image_id=315680d7-c2d3-4042-8d33-632d758e85f2 2023-12-02T22:50:20.232 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T22:50:20.232 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:50:20.233 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-02T22:50:20.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T22:50:20.233 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T22:50:20.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T22:50:20.236 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-12-02T22:50:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-02T22:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-02T22:50:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_27 2023-12-02T22:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:50:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-02T22:50:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-02T22:50:21.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:50:21.437+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324aa17a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:21.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:50:21.437+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a699180 [3/cba1b5db-1e9a-4652-8e1d-956d8a38b3c8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-02T22:50:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-02T22:50:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:50:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_28 2023-12-02T22:50:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:50:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-02T22:50:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-02T22:50:24.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:24.444+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff14cd69c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:24.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:24.444+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff15543680 [3/65974775-a20e-4c7a-9d15-5a68ebf338c6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-02T22:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_29 2023-12-02T22:50:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:50:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-02T22:50:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-02T22:50:27.327 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:50:27.326+0000 7eff3a87e640 -1 reset not still connected to 0x559717b59860 2023-12-02T22:50:27.328 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:50:27.326+0000 7eff3a87e640 -1 reset not still connected to 0x5597183a21a0 2023-12-02T22:50:27.328 INFO:tasks.ceph.cluster1.osd.2.smithi012.stderr:2023-12-02T22:50:27.326+0000 7eff3a87e640 -1 reset not still connected to 0x55971877f110 2023-12-02T22:50:27.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:27.372+0000 7faf204d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee405929c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:27.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:27.372+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40274000 [3/8686cb37-c0f3-4cb1-bfb7-7d89f4ef29c2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-02T22:50:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-02T22:50:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:50:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_30 2023-12-02T22:50:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:50:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-02T22:50:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-02T22:50:30.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:30.385+0000 7f873553c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c457040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:30.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:30.385+0000 7f873bd49640 -1 rbd::mirror::ImageReplayer: 0x55f34bae1180 [3/af5e53fb-d079-4e24-8f20-85b325a485b5] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-02T22:50:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-02T22:50:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_31 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:50:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-02T22:50:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-02T22:50:33.335 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:50:33.334+0000 7f2e21bfd640 -1 reset not still connected to 0x557cf4f54410 2023-12-02T22:50:33.336 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:50:33.335+0000 7f2e21bfd640 -1 reset not still connected to 0x557d05c488f0 2023-12-02T22:50:33.337 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T22:50:33.335+0000 7f2e21bfd640 -1 reset not still connected to 0x557d1a83d110 2023-12-02T22:50:33.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:33.396+0000 7f8734d3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c3d64e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:33.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:33.396+0000 7f873bd49640 -1 rbd::mirror::ImageReplayer: 0x55f34bae0a00 [3/93fcda49-830e-47ed-9f3a-10f4ff483bab] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2023-12-02T22:50:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-02T22:50:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:50:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-02T22:50:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-02T22:50:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:50:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_32 2023-12-02T22:50:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:50:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-02T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-02T22:50:36.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:36.396+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff158b0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:36.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:36.396+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff15543900 [3/662b719c-5386-4ec4-8056-aa4e67a37e59] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:50:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-02T22:50:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-02T22:50:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:50:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_33 2023-12-02T22:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-02T22:50:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-02T22:50:39.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:39.414+0000 7f873553c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c5e4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:39.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:39.414+0000 7f873bd49640 -1 rbd::mirror::ImageReplayer: 0x55f34bae1900 [3/d64bed83-0a8b-46ea-b180-1169dba0dac7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-02T22:50:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-02T22:50:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:50:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_34 2023-12-02T22:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-02T22:50:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-02T22:50:42.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:42.406+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff14cd6d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:42.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:42.406+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff15542f00 [3/72af8c9e-08e6-4b8b-a6d4-3048fd53a1e1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2023-12-02T22:50:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-02T22:50:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:50:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-02T22:50:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-02T22:50:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:50:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_35 2023-12-02T22:50:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:50:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-02T22:50:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-02T22:50:45.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:45.381+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff14cd6ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:45.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:45.381+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff15542a00 [3/4bb9743c-64d8-40e8-b02a-6a91310c2444] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2023-12-02T22:50:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-02T22:50:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:50:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-02T22:50:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-02T22:50:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:50:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_36 2023-12-02T22:50:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:50:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-02T22:50:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-02T22:50:48.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:48.641+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff149f8820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:48.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:50:48.641+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff15543b80 [3/4b378cd6-dd34-4f37-b8aa-ec8c5a2768b0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-02T22:50:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-02T22:50:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:50:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_37 2023-12-02T22:50:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:50:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-02T22:50:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-02T22:50:51.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:51.430+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40149ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:51.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:50:51.430+0000 7faf26ce2640 -1 rbd::mirror::ImageReplayer: 0x55ee40434c80 [3/6de1f79c-583e-4974-ae84-18b971385b78] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-02T22:50:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-02T22:50:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:50:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_38 2023-12-02T22:50:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:50:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-02T22:50:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-02T22:50:54.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:54.456+0000 7f8734d3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c477380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:54.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:50:54.456+0000 7f873b548640 -1 rbd::mirror::ImageReplayer: 0x55f34bae0500 [3/8dedd0af-fa04-42db-9022-5c3e623cc7a9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-02T22:50:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:50:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-02T22:50:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:50:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:50:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:50:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_39 2023-12-02T22:50:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:50:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-02T22:50:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-02T22:50:57.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:50:57.376+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab69040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:50:57.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:50:57.376+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a698c80 [3/e5ea4907-8178-4427-a782-ee8006b75e82] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:50:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:50:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:50:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:50:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2023-12-02T22:50:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-02T22:50:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:50:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:50:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:50:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:50:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-02T22:51:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-02T22:51:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:51:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_40 2023-12-02T22:51:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:51:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-02T22:51:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-02T22:51:00.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:51:00.386+0000 7f873553c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c477ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:00.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:51:00.386+0000 7f873b548640 -1 rbd::mirror::ImageReplayer: 0x55f34bae0280 [3/7f8c70ee-6301-46d2-8c34-89c793426067] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-02T22:51:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-02T22:51:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_41 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-02T22:51:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-02T22:51:03.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:51:03.403+0000 7f8734d3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f34c477860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:03.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:51:03.403+0000 7f873b548640 -1 rbd::mirror::ImageReplayer: 0x55f34bae1400 [3/a49a8b18-32ec-4784-9e10-57e3c1d4cf26] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-02T22:51:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_42 2023-12-02T22:51:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:51:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-02T22:51:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-02T22:51:06.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:51:06.408+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40593380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:06.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:51:06.408+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40434f00 [3/7d8c8ff7-9e39-4fd4-a378-75b3505c1c8c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:51:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-02T22:51:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-02T22:51:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:51:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_43 2023-12-02T22:51:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:51:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-02T22:51:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-02T22:51:09.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:09.411+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab23040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:09.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:09.411+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54b400 [3/dabad757-1fd3-41ef-b3ef-13b1187bb4f2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-02T22:51:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-02T22:51:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:51:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_44 2023-12-02T22:51:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:51:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-02T22:51:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-02T22:51:12.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:12.424+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab23520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:12.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:12.424+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54a280 [3/a442254e-d4a3-4fcf-9c21-028b8a99b2f0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-02T22:51:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-02T22:51:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:51:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_45 2023-12-02T22:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-02T22:51:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-02T22:51:15.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:51:15.387+0000 7faf204d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40593520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:15.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:51:15.387+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40435680 [3/09f87dc1-458a-4e3a-a6c6-7afee7b3129e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2023-12-02T22:51:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-02T22:51:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:51:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-02T22:51:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-02T22:51:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:51:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_46 2023-12-02T22:51:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:51:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-02T22:51:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-02T22:51:18.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:18.439+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab23a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:18.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:18.439+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54b900 [3/c1d2a65d-c465-4d9a-8cce-99526d99432c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-02T22:51:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-02T22:51:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:51:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_47 2023-12-02T22:51:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:51:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-02T22:51:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-02T22:51:21.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:51:21.443+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff1595b6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:21.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:51:21.443+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff1531e000 [3/9431194f-1321-4cad-a347-45f6490d98bd] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2023-12-02T22:51:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-02T22:51:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:51:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-02T22:51:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-02T22:51:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:51:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_48 2023-12-02T22:51:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:51:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-02T22:51:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-02T22:51:24.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:24.445+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab541a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:24.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:24.445+0000 7f6bdd535640 -1 rbd::mirror::ImageReplayer: 0x56324a54a500 [3/e848264b-794b-4557-91db-d8f1e8b231f2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-02T22:51:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-02T22:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_49 2023-12-02T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:51:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-02T22:51:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-02T22:51:27.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:27.385+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab54340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:27.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:27.385+0000 7f6bdd535640 -1 rbd::mirror::ImageReplayer: 0x56324a698780 [3/8c19922c-ba85-47c0-866f-53244088c863] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2023-12-02T22:51:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-02T22:51:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:51:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2023-12-02T22:51:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-02T22:51:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2023-12-02T22:51:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-02T22:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:51:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_50 2023-12-02T22:51:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:51:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-02T22:51:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-02T22:51:30.338 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5821163920400915) > weight (0.25) 2023-12-02T22:51:30.338 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T22:51:30.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T22:51:30.338 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-02T22:51:30.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:51:30.339+0000 7f873c54a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 116892) UID: 1000 2023-12-02T22:51:30.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:51:30.402+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee406ca000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:30.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:51:30.402+0000 7faf264e1640 -1 rbd::mirror::ImageReplayer: 0x55ee40434500 [3/4b794018-5af4-4f87-a8d7-86141ee6bc32] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-02T22:51:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: image deletions should propagate' 2023-12-02T22:51:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:51:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:51:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:51:32 TEST: image deletions should propagate 2023-12-02T22:51:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:51:32 TEST: image deletions should propagate 2023-12-02T22:51:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-02T22:51:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_count=0 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local count 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_count=0 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:51:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2023-12-02T22:51:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T22:51:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'images: ' 2023-12-02T22:51:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 2 2023-12-02T22:51:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ count=0 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state 2023-12-02T22:51:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2023-12-02T22:51:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2023-12-02T22:51:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2023-12-02T22:51:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T22:51:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2023-12-02T22:51:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2023-12-02T22:51:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T22:51:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=OK 2023-12-02T22:51:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = ERROR 2023-12-02T22:51:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = OK 2023-12-02T22:51:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2023-12-02T22:51:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2023-12-02T22:51:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-02T22:51:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-02T22:51:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-02T22:51:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2023-12-02T22:51:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-02T22:51:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-02T22:51:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2023-12-02T22:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2023-12-02T22:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-02T22:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-02T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-02T22:51:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-02T22:51:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:51:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-02T22:51:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:51:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:51:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:51:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-02T22:51:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2023-12-02T22:51:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2023-12-02T22:51:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-02T22:51:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-02T22:51:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-02T22:51:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-02T22:51:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-02T22:51:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2023-12-02T22:51:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2023-12-02T22:51:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-02T22:51:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-02T22:51:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2023-12-02T22:51:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2023-12-02T22:51:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-02T22:51:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-02T22:51:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2023-12-02T22:51:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-02T22:51:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-02T22:51:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2023-12-02T22:51:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2023-12-02T22:51:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-02T22:51:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-02T22:51:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:51:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-02T22:51:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:51:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-02T22:51:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-02T22:51:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-02T22:51:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2023-12-02T22:51:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2023-12-02T22:51:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-02T22:51:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:51:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-02T22:51:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-02T22:51:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-02T22:51:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:51:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-02T22:51:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-02T22:51:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2023-12-02T22:51:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-02T22:51:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-02T22:51:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_11 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-02T22:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-02T22:51:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-12-02T22:51:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_11 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-02T22:51:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-02T22:51:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_12 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-02T22:51:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-02T22:51:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_12 2023-12-02T22:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2023-12-02T22:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-02T22:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-02T22:51:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_13 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2023-12-02T22:51:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-02T22:51:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-02T22:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-12-02T22:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2023-12-02T22:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_13 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-02T22:51:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_14 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-02T22:51:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-02T22:51:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-12-02T22:51:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_14 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-02T22:51:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_15 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-02T22:51:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-02T22:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-12-02T22:51:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2023-12-02T22:51:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_15 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-02T22:51:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:51:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_16 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-02T22:51:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-02T22:51:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-12-02T22:51:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2023-12-02T22:51:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_16 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-02T22:51:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-02T22:51:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_17 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2023-12-02T22:51:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-02T22:51:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-02T22:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-12-02T22:51:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_17 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-02T22:51:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-02T22:51:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_18 2023-12-02T22:51:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2023-12-02T22:51:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-02T22:51:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-02T22:51:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_18 2023-12-02T22:51:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2023-12-02T22:51:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-02T22:51:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-02T22:51:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_19 2023-12-02T22:51:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2023-12-02T22:51:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-02T22:51:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-02T22:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_19 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2023-12-02T22:51:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-02T22:51:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-02T22:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_20 2023-12-02T22:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2023-12-02T22:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-02T22:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-02T22:51:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_20 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2023-12-02T22:51:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-02T22:51:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-02T22:51:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_21 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-02T22:51:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-02T22:51:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_21 2023-12-02T22:51:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2023-12-02T22:51:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-02T22:51:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-02T22:51:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_22 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-02T22:51:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-02T22:51:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_22 2023-12-02T22:51:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2023-12-02T22:51:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-02T22:51:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-02T22:51:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_23 2023-12-02T22:51:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2023-12-02T22:51:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-02T22:51:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-02T22:51:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-12-02T22:51:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2023-12-02T22:51:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_23 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-02T22:51:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-02T22:51:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2023-12-02T22:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_24 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2023-12-02T22:51:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-02T22:51:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-02T22:51:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_24 2023-12-02T22:51:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2023-12-02T22:51:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-02T22:51:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-02T22:51:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:51:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_25 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-02T22:51:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-02T22:51:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-12-02T22:51:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2023-12-02T22:51:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_25 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-02T22:51:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-02T22:51:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_26 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-02T22:51:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-02T22:51:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_26 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2023-12-02T22:51:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-02T22:51:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_27 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-02T22:51:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-02T22:51:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-12-02T22:51:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2023-12-02T22:51:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_27 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-02T22:51:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:51:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_28 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-02T22:51:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-02T22:51:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_28 2023-12-02T22:51:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2023-12-02T22:51:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-02T22:51:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-02T22:51:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_29 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-02T22:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-02T22:51:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-12-02T22:51:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2023-12-02T22:51:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_29 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-02T22:51:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2023-12-02T22:51:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_30 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-02T22:51:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-02T22:51:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_30 2023-12-02T22:51:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2023-12-02T22:51:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-02T22:51:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-02T22:51:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_31 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2023-12-02T22:51:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-02T22:51:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-02T22:51:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_31 2023-12-02T22:51:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2023-12-02T22:51:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-02T22:51:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-02T22:51:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_32 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-02T22:51:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-02T22:51:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_32 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-02T22:51:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-02T22:51:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_33 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-02T22:51:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-02T22:51:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_33 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2023-12-02T22:51:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-02T22:51:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.339 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T22:51:38.339 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:51:38.339 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T22:51:38.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T22:51:38.340 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T22:51:38.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T22:51:38.342 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-02T22:51:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_34 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-02T22:51:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-02T22:51:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_34 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-02T22:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-02T22:51:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_35 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-02T22:51:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-02T22:51:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_35 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2023-12-02T22:51:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-02T22:51:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:51:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_36 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-02T22:51:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-02T22:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-12-02T22:51:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_36 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-02T22:51:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-02T22:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_37 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2023-12-02T22:51:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-02T22:51:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-02T22:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-12-02T22:51:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_37 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-02T22:51:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-02T22:51:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_38 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-02T22:51:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-02T22:51:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-12-02T22:51:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_38 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-02T22:51:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-02T22:51:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_39 2023-12-02T22:51:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2023-12-02T22:51:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-02T22:51:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-02T22:51:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-12-02T22:51:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_39 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-02T22:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_40 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-02T22:51:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-02T22:51:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-12-02T22:51:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_40 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-02T22:51:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_41 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-02T22:51:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-02T22:51:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_41 2023-12-02T22:51:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2023-12-02T22:51:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-02T22:51:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-02T22:51:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_42 2023-12-02T22:51:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2023-12-02T22:51:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-02T22:51:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-02T22:51:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_42 2023-12-02T22:51:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2023-12-02T22:51:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-02T22:51:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-02T22:51:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_43 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-02T22:51:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-02T22:51:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_43 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-02T22:51:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_44 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-02T22:51:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-02T22:51:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_44 2023-12-02T22:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2023-12-02T22:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-02T22:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_45 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-02T22:51:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-02T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_45 2023-12-02T22:51:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2023-12-02T22:51:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-02T22:51:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-02T22:51:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_46 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-02T22:51:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-02T22:51:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_46 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-02T22:51:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-02T22:51:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_47 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-02T22:51:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-02T22:51:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_47 2023-12-02T22:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2023-12-02T22:51:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-02T22:51:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-02T22:51:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_48 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-02T22:51:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-02T22:51:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_48 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-02T22:51:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-02T22:51:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_49 2023-12-02T22:51:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2023-12-02T22:51:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-02T22:51:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-02T22:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_49 2023-12-02T22:51:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2023-12-02T22:51:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-02T22:51:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-02T22:51:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_50 2023-12-02T22:51:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2023-12-02T22:51:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-02T22:51:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-02T22:51:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:51:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_50 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2023-12-02T22:51:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-02T22:51:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:51:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:51:41 TEST: delete images during bootstrap 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:51:41 TEST: delete images during bootstrap 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=image 2023-12-02T22:51:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-02T22:51:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd: mirroring is already configured for image mode 2023-12-02T22:51:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-02T22:51:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=image 2023-12-02T22:51:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-02T22:51:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd: mirroring is already configured for image mode 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2023-12-02T22:51:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=cluster1 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 = cluster1 ']' 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=test 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-02T22:51:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:51:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-02T22:51:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-02T22:51:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:51:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-02T22:51:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:51:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_1': 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 18d3e0d4232a 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.18d3e0d4232a 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:51:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:51:41 2023 2023-12-02T22:51:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:51:41 2023 2023-12-02T22:51:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:51:41 2023 2023-12-02T22:51:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:51:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:51:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 75cdc508-0a5b-4fa3-9c74-7be4894623e7 2023-12-02T22:51:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:51:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-02T22:51:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:51:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2023-12-02T22:51:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:51:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:51:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 10087424 bytes 2023-12-02T22:51:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 28362752 bytes 2023-12-02T22:51:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 93344768 bytes 2023-12-02T22:51:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 126194688 bytes 2023-12-02T22:51:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 164941824 bytes 2023-12-02T22:51:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 175847424 bytes 2023-12-02T22:51:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 179446784 bytes 2023-12-02T22:51:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 183333888 bytes 2023-12-02T22:51:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 188559360 bytes 2023-12-02T22:51:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 213804032 bytes 2023-12-02T22:51:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 236609536 bytes 2023-12-02T22:51:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 256651264 bytes 2023-12-02T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 290781184 bytes 2023-12-02T22:51:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 359612416 bytes 2023-12-02T22:51:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 417927168 bytes 2023-12-02T22:51:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 478814208 bytes 2023-12-02T22:51:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 481319936 bytes 2023-12-02T22:51:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 567453696 bytes 2023-12-02T22:51:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 575749120 bytes 2023-12-02T22:51:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 695231488 bytes 2023-12-02T22:51:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 774983680 bytes 2023-12-02T22:51:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 805388288 bytes 2023-12-02T22:51:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 817184768 bytes 2023-12-02T22:51:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 850560000 bytes 2023-12-02T22:51:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 924328960 bytes 2023-12-02T22:51:45.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1005093888 bytes 2023-12-02T22:51:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1062976512 bytes 2023-12-02T22:51:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1128998912 bytes 2023-12-02T22:51:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1151451136 bytes 2023-12-02T22:51:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1209649152 bytes 2023-12-02T22:51:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1293787136 bytes 2023-12-02T22:51:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1332844544 bytes 2023-12-02T22:51:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1415526400 bytes 2023-12-02T22:51:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1522600960 bytes 2023-12-02T22:51:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1552539648 bytes 2023-12-02T22:51:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1555477504 bytes 2023-12-02T22:51:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1628221440 bytes 2023-12-02T22:51:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1685996544 bytes 2023-12-02T22:51:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1737896960 bytes 2023-12-02T22:51:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1795220480 bytes 2023-12-02T22:51:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1860569088 bytes 2023-12-02T22:51:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1918593024 bytes 2023-12-02T22:51:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:51:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:51:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:51:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2023-12-02T22:51:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-02T22:51:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12a1d4e3b984 2023-12-02T22:51:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:51:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_1 present 12a1d4e3b984 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1d4e3b984 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:51:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:51:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2023-12-02T22:51:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:51:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:51:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2023-12-02T22:51:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-02T22:51:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:51:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12a1d4e3b984 2023-12-02T22:51:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12a1d4e3b984 2023-12-02T22:51:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12a1d4e3b984 -o 12a1d4e3b984 = 12a1d4e3b984 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:51:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-02T22:51:50.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:50.552+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab54680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:51:50.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:51:50.552+0000 7f6bdd535640 -1 rbd::mirror::ImageReplayer: 0x56324a54b900 [3/75cdc508-0a5b-4fa3-9c74-7be4894623e7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:51:53.237 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6122678019370341) > weight (0.25) 2023-12-02T22:51:53.237 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9530793179107901) > weight (0.25) 2023-12-02T22:51:53.237 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T22:51:53.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T22:51:53.238 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T22:51:53.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:51:53.237+0000 7f559e7b0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 118549) UID: 1000 2023-12-02T22:52:00.239 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T22:52:00.239 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:52:00.239 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-02T22:52:00.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T22:52:00.240 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T22:52:00.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T22:52:00.243 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-02T22:52:18.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:52:18.846+0000 7fbc4eb10640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e61e657c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-02T22:52:18.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:52:18.846+0000 7fbc4eb10640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e61e657c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-02T22:52:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-02T22:52:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-02T22:52:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:52:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-02T22:52:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:52:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:52:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:52:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-02T22:52:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-02T22:52:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:52:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:52:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-02T22:52:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-02T22:52:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:52:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:52:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:52:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_1 2023-12-02T22:52:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2023-12-02T22:52:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-02T22:52:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-02T22:52:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:52:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:52:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:52:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2023-12-02T22:52:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-02T22:52:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:52:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:52:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:52:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:52:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:52:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:52:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:52:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:52:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-02T22:52:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-02T22:52:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:52:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:52:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-02T22:52:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:52:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-02T22:52:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_2': 2023-12-02T22:52:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 18f1d01a8c3a 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.18f1d01a8c3a 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:52:19 2023 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:52:19 2023 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:52:19 2023 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:52:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:52:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 1944b265-5dfa-4199-9620-cbbcc20564da 2023-12-02T22:52:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:52:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-02T22:52:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:52:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:52:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2023-12-02T22:52:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:52:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:52:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 51705856 bytes 2023-12-02T22:52:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 124110848 bytes 2023-12-02T22:52:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 176183296 bytes 2023-12-02T22:52:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 283833344 bytes 2023-12-02T22:52:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 323492864 bytes 2023-12-02T22:52:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 338664448 bytes 2023-12-02T22:52:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 350401536 bytes 2023-12-02T22:52:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 374100992 bytes 2023-12-02T22:52:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 420060160 bytes 2023-12-02T22:52:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 456963072 bytes 2023-12-02T22:52:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 482164736 bytes 2023-12-02T22:52:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 584956928 bytes 2023-12-02T22:52:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 638505984 bytes 2023-12-02T22:52:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 652019712 bytes 2023-12-02T22:52:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 706495488 bytes 2023-12-02T22:52:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 719360000 bytes 2023-12-02T22:52:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 730449920 bytes 2023-12-02T22:52:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 763115520 bytes 2023-12-02T22:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 861548544 bytes 2023-12-02T22:52:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 902072320 bytes 2023-12-02T22:52:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 963732480 bytes 2023-12-02T22:52:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1008153600 bytes 2023-12-02T22:52:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1030004736 bytes 2023-12-02T22:52:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1070282752 bytes 2023-12-02T22:52:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1096297472 bytes 2023-12-02T22:52:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1096908800 bytes 2023-12-02T22:52:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1109864448 bytes 2023-12-02T22:52:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1129977856 bytes 2023-12-02T22:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1144528896 bytes 2023-12-02T22:52:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1212259328 bytes 2023-12-02T22:52:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1251881984 bytes 2023-12-02T22:52:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1281916928 bytes 2023-12-02T22:52:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1350221824 bytes 2023-12-02T22:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1350844416 bytes 2023-12-02T22:52:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1393725440 bytes 2023-12-02T22:52:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1536986112 bytes 2023-12-02T22:52:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1567296512 bytes 2023-12-02T22:52:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1620501504 bytes 2023-12-02T22:52:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1695601664 bytes 2023-12-02T22:52:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1703462912 bytes 2023-12-02T22:52:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1791582208 bytes 2023-12-02T22:52:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1849661440 bytes 2023-12-02T22:52:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1850293248 bytes 2023-12-02T22:52:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1870841856 bytes 2023-12-02T22:52:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1916058624 bytes 2023-12-02T22:52:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1967374336 bytes 2023-12-02T22:52:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2045824000 bytes 2023-12-02T22:52:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2103793664 bytes 2023-12-02T22:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2182370304 bytes 2023-12-02T22:52:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2275389440 bytes 2023-12-02T22:52:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2307817472 bytes 2023-12-02T22:52:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2359702528 bytes 2023-12-02T22:52:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2365031424 bytes 2023-12-02T22:52:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2422491136 bytes 2023-12-02T22:52:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2526457856 bytes 2023-12-02T22:52:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2549734400 bytes 2023-12-02T22:52:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2576533504 bytes 2023-12-02T22:52:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2592992256 bytes 2023-12-02T22:52:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2624123904 bytes 2023-12-02T22:52:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2663556096 bytes 2023-12-02T22:52:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2738196480 bytes 2023-12-02T22:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2778389504 bytes 2023-12-02T22:52:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2820598784 bytes 2023-12-02T22:52:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2911803392 bytes 2023-12-02T22:52:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2931947520 bytes 2023-12-02T22:52:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 3062747136 bytes 2023-12-02T22:52:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3107206144 bytes 2023-12-02T22:52:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3127970816 bytes 2023-12-02T22:52:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3135083520 bytes 2023-12-02T22:52:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3154793472 bytes 2023-12-02T22:52:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3175517184 bytes 2023-12-02T22:52:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3256166400 bytes 2023-12-02T22:52:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3333846016 bytes 2023-12-02T22:52:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3371330560 bytes 2023-12-02T22:52:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3409501184 bytes 2023-12-02T22:52:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3447240704 bytes 2023-12-02T22:52:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3501889536 bytes 2023-12-02T22:52:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3560809472 bytes 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-02T22:52:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:52:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12c266e9dee0 2023-12-02T22:52:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:52:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_2 present 12c266e9dee0 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c266e9dee0 2023-12-02T22:52:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-02T22:52:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:52:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c266e9dee0 2023-12-02T22:52:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c266e9dee0 2023-12-02T22:52:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c266e9dee0 -o 12c266e9dee0 = 12c266e9dee0 2023-12-02T22:52:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:52:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:52:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:52:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-02T22:52:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:52:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:52:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:52:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-02T22:52:40.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:52:40.448+0000 7f20d931f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff1595bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:52:40.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:52:40.448+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff1531e000 [3/1944b265-5dfa-4199-9620-cbbcc20564da] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:53:02.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:53:02.244+0000 7fd85d134640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e3358f7400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-02T22:53:02.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:53:02.244+0000 7fd85d134640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e3358f7400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-02T22:53:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:53:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-02T22:53:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-02T22:53:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-02T22:53:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:53:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-02T22:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-02T22:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:53:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-02T22:53:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-02T22:53:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:53:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:53:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:53:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_2 2023-12-02T22:53:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2023-12-02T22:53:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-02T22:53:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-02T22:53:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:53:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:53:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:53:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:53:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-02T22:53:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-02T22:53:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:53:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-02T22:53:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:53:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-02T22:53:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_3': 2023-12-02T22:53:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:53:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 19097cbf5273 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.19097cbf5273 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:53:03 2023 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:53:03 2023 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:53:03 2023 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 0da11d89-5775-46ec-bca6-f9404ace7f73 2023-12-02T22:53:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:53:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-02T22:53:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:53:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2023-12-02T22:53:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:53:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:53:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 28313600 bytes 2023-12-02T22:53:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 71666688 bytes 2023-12-02T22:53:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 141499392 bytes 2023-12-02T22:53:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 162201600 bytes 2023-12-02T22:53:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 168968192 bytes 2023-12-02T22:53:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 186723328 bytes 2023-12-02T22:53:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 225613824 bytes 2023-12-02T22:53:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 262112256 bytes 2023-12-02T22:53:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 287775744 bytes 2023-12-02T22:53:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 351273984 bytes 2023-12-02T22:53:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 379895808 bytes 2023-12-02T22:53:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 445377536 bytes 2023-12-02T22:53:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 495963136 bytes 2023-12-02T22:53:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 517449728 bytes 2023-12-02T22:53:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 585102336 bytes 2023-12-02T22:53:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 598208512 bytes 2023-12-02T22:53:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 605946880 bytes 2023-12-02T22:53:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 644136960 bytes 2023-12-02T22:53:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 743073792 bytes 2023-12-02T22:53:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 778702848 bytes 2023-12-02T22:53:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 819315712 bytes 2023-12-02T22:53:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 905220096 bytes 2023-12-02T22:53:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1001230336 bytes 2023-12-02T22:53:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1001820160 bytes 2023-12-02T22:53:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1078358016 bytes 2023-12-02T22:53:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1146602496 bytes 2023-12-02T22:53:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1232325632 bytes 2023-12-02T22:53:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1238277120 bytes 2023-12-02T22:53:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1260599296 bytes 2023-12-02T22:53:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1346802688 bytes 2023-12-02T22:53:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1358547968 bytes 2023-12-02T22:53:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1425965056 bytes 2023-12-02T22:53:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1452090368 bytes 2023-12-02T22:53:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1502162944 bytes 2023-12-02T22:53:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1530203136 bytes 2023-12-02T22:53:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1576548352 bytes 2023-12-02T22:53:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1594408960 bytes 2023-12-02T22:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1667401728 bytes 2023-12-02T22:53:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1734081536 bytes 2023-12-02T22:53:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1788928000 bytes 2023-12-02T22:53:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1813134336 bytes 2023-12-02T22:53:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1845095424 bytes 2023-12-02T22:53:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1909405696 bytes 2023-12-02T22:53:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1937734656 bytes 2023-12-02T22:53:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2006790144 bytes 2023-12-02T22:53:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2057512960 bytes 2023-12-02T22:53:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2073524224 bytes 2023-12-02T22:53:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2077607936 bytes 2023-12-02T22:53:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2119581696 bytes 2023-12-02T22:53:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2177526784 bytes 2023-12-02T22:53:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2205409280 bytes 2023-12-02T22:53:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2261810176 bytes 2023-12-02T22:53:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2304481280 bytes 2023-12-02T22:53:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2373987328 bytes 2023-12-02T22:53:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2412527616 bytes 2023-12-02T22:53:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2435716096 bytes 2023-12-02T22:53:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2501165056 bytes 2023-12-02T22:53:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2542022656 bytes 2023-12-02T22:53:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2560321536 bytes 2023-12-02T22:53:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2572063744 bytes 2023-12-02T22:53:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2626682880 bytes 2023-12-02T22:53:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2650296320 bytes 2023-12-02T22:53:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2670974976 bytes 2023-12-02T22:53:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2704505856 bytes 2023-12-02T22:53:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2791764992 bytes 2023-12-02T22:53:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2822682624 bytes 2023-12-02T22:53:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2925261824 bytes 2023-12-02T22:53:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2971458560 bytes 2023-12-02T22:53:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2980826112 bytes 2023-12-02T22:53:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3015696384 bytes 2023-12-02T22:53:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3058888704 bytes 2023-12-02T22:53:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3121418240 bytes 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12a1debee41b 2023-12-02T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:53:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 present 12a1debee41b 2023-12-02T22:53:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1debee41b 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12a1debee41b 2023-12-02T22:53:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12a1debee41b 2023-12-02T22:53:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12a1debee41b -o 12a1debee41b = 12a1debee41b 2023-12-02T22:53:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:53:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:53:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:53:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-02T22:53:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-02T22:53:14.342 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4154775357255941) > weight (0.25) 2023-12-02T22:53:14.343 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9397128746616703) > weight (0.25) 2023-12-02T22:53:14.343 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8349014418624254) > weight (0.25) 2023-12-02T22:53:14.343 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6906406675809353) > weight (0.25) 2023-12-02T22:53:14.343 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-02T22:53:18.245 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3099741405951163) > weight (0.25) 2023-12-02T22:53:18.245 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-02T22:53:18.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-02T22:53:18.245 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T22:53:18.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:53:18.245+0000 7fbc5cb2c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 100679) UID: 1000 2023-12-02T22:53:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:53:20.552+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324aabc820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:53:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:53:20.552+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a698780 [3/0da11d89-5775-46ec-bca6-f9404ace7f73] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:53:25.246 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-02T22:53:25.247 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:53:25.247 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-02T22:53:25.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T22:53:25.247 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T22:53:25.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T22:53:25.250 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-02T22:53:43.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:53:43.522+0000 7fd86a14e640 -1 rbd::mirror::ImageMap: 0x55e334db4ea0 handle_peer_ack_remove: failed to remove global_image_id=0da11d89-5775-46ec-bca6-f9404ace7f73 2023-12-02T22:53:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:53:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-02T22:53:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12a1debee41b 2023-12-02T22:53:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted 12a1debee41b 2023-12-02T22:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1debee41b 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12a1debee41b 2023-12-02T22:53:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12a1debee41b 2023-12-02T22:53:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12a1debee41b -o 12a1debee41b = 12a1debee41b 2023-12-02T22:53:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:53:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:53:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted 12a1debee41b 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1debee41b 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12a1debee41b 2023-12-02T22:53:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12a1debee41b 2023-12-02T22:53:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12a1debee41b -o 12a1debee41b = 12a1debee41b 2023-12-02T22:53:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:53:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:53:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:53:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted 12a1debee41b 2023-12-02T22:53:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:53:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1debee41b 2023-12-02T22:53:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12a1debee41b 2023-12-02T22:53:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12a1debee41b 2023-12-02T22:53:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12a1debee41b -o 12a1debee41b = 12a1debee41b 2023-12-02T22:53:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:53:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:53:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 4 2023-12-02T22:53:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted 12a1debee41b 2023-12-02T22:53:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:53:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1debee41b 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12a1debee41b 2023-12-02T22:53:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12a1debee41b 2023-12-02T22:53:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12a1debee41b -o 12a1debee41b = 12a1debee41b 2023-12-02T22:53:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:53:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:53:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:53:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 8 2023-12-02T22:53:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted 12a1debee41b 2023-12-02T22:53:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:53:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12a1debee41b 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-02T22:53:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:53:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-02T22:53:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:53:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:53:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:53:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:53:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-02T22:53:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-02T22:53:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-02T22:53:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:53:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:53:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_3 2023-12-02T22:53:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2023-12-02T22:53:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-02T22:53:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-02T22:53:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:53:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-02T22:53:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-02T22:53:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:53:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:53:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-02T22:53:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:53:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_4': 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1927b986a4f9 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1927b986a4f9 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:53:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:53:59 2023 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:53:59 2023 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:53:59 2023 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c9e0dd40-1943-4857-b1c4-3058586833b7 2023-12-02T22:53:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:53:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-02T22:53:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:53:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:53:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:53:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:53:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:53:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:53:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:54:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 33007616 bytes 2023-12-02T22:54:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 123283456 bytes 2023-12-02T22:54:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 205520896 bytes 2023-12-02T22:54:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 223080448 bytes 2023-12-02T22:54:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 223719424 bytes 2023-12-02T22:54:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 234706944 bytes 2023-12-02T22:54:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 248144896 bytes 2023-12-02T22:54:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 278848512 bytes 2023-12-02T22:54:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 284914688 bytes 2023-12-02T22:54:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 315607040 bytes 2023-12-02T22:54:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 369103872 bytes 2023-12-02T22:54:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 432699392 bytes 2023-12-02T22:54:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 474851328 bytes 2023-12-02T22:54:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 498830336 bytes 2023-12-02T22:54:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 572060672 bytes 2023-12-02T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 605539328 bytes 2023-12-02T22:54:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 656530432 bytes 2023-12-02T22:54:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 693961728 bytes 2023-12-02T22:54:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 713783296 bytes 2023-12-02T22:54:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 765102080 bytes 2023-12-02T22:54:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 831219712 bytes 2023-12-02T22:54:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 888779776 bytes 2023-12-02T22:54:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 929532928 bytes 2023-12-02T22:54:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 978414592 bytes 2023-12-02T22:54:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1028320256 bytes 2023-12-02T22:54:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1085967360 bytes 2023-12-02T22:54:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1142495232 bytes 2023-12-02T22:54:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1195565056 bytes 2023-12-02T22:54:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1215129600 bytes 2023-12-02T22:54:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1234765824 bytes 2023-12-02T22:54:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1266992128 bytes 2023-12-02T22:54:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1333992448 bytes 2023-12-02T22:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1367731200 bytes 2023-12-02T22:54:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1382471680 bytes 2023-12-02T22:54:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1473580032 bytes 2023-12-02T22:54:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1539565568 bytes 2023-12-02T22:54:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1602024448 bytes 2023-12-02T22:54:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1628617728 bytes 2023-12-02T22:54:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1655832576 bytes 2023-12-02T22:54:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1676324864 bytes 2023-12-02T22:54:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1771071488 bytes 2023-12-02T22:54:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1803534336 bytes 2023-12-02T22:54:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1833435136 bytes 2023-12-02T22:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1886021632 bytes 2023-12-02T22:54:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1983568896 bytes 2023-12-02T22:54:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1984194560 bytes 2023-12-02T22:54:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2013897728 bytes 2023-12-02T22:54:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2065145856 bytes 2023-12-02T22:54:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2102530048 bytes 2023-12-02T22:54:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2179381248 bytes 2023-12-02T22:54:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2231664640 bytes 2023-12-02T22:54:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2256646144 bytes 2023-12-02T22:54:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2292636672 bytes 2023-12-02T22:54:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2320842752 bytes 2023-12-02T22:54:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2331063296 bytes 2023-12-02T22:54:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2354563072 bytes 2023-12-02T22:54:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2377743360 bytes 2023-12-02T22:54:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2436731904 bytes 2023-12-02T22:54:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2455040000 bytes 2023-12-02T22:54:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2455643136 bytes 2023-12-02T22:54:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2534949888 bytes 2023-12-02T22:54:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2617958400 bytes 2023-12-02T22:54:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2618560512 bytes 2023-12-02T22:54:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2644788224 bytes 2023-12-02T22:54:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2686611456 bytes 2023-12-02T22:54:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2770501632 bytes 2023-12-02T22:54:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2794669056 bytes 2023-12-02T22:54:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2870367232 bytes 2023-12-02T22:54:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2893533184 bytes 2023-12-02T22:54:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2919877632 bytes 2023-12-02T22:54:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2925968384 bytes 2023-12-02T22:54:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2986201088 bytes 2023-12-02T22:54:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3054817280 bytes 2023-12-02T22:54:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3068063744 bytes 2023-12-02T22:54:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3110595584 bytes 2023-12-02T22:54:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3116979200 bytes 2023-12-02T22:54:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3143715840 bytes 2023-12-02T22:54:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3176088576 bytes 2023-12-02T22:54:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3188379648 bytes 2023-12-02T22:54:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3198184448 bytes 2023-12-02T22:54:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3250096128 bytes 2023-12-02T22:54:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3383312384 bytes 2023-12-02T22:54:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3425814528 bytes 2023-12-02T22:54:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3470172160 bytes 2023-12-02T22:54:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3513448448 bytes 2023-12-02T22:54:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3524356096 bytes 2023-12-02T22:54:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3598073856 bytes 2023-12-02T22:54:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3678625792 bytes 2023-12-02T22:54:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3702823936 bytes 2023-12-02T22:54:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:54:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:54:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:54:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:54:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-02T22:54:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-02T22:54:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:54:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12c21d784521 2023-12-02T22:54:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:54:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_4 present 12c21d784521 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c21d784521 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:54:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2023-12-02T22:54:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-02T22:54:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:54:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c21d784521 2023-12-02T22:54:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c21d784521 2023-12-02T22:54:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c21d784521 -o 12c21d784521 = 12c21d784521 2023-12-02T22:54:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:54:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:54:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:54:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-02T22:54:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-02T22:54:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:54:20.553+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324aa17d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:54:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:54:20.553+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54a500 [3/c9e0dd40-1943-4857-b1c4-3058586833b7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:54:41.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:54:41.881+0000 7fd85d935640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e335b99800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-02T22:54:41.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:54:41.881+0000 7fd85d935640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e335b99800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-02T22:54:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:54:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-02T22:54:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:54:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2023-12-02T22:54:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-02T22:54:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:54:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-02T22:54:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:54:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-02T22:54:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:54:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-02T22:54:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:54:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-02T22:54:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:54:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:54:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:54:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:54:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-02T22:54:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-02T22:54:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-02T22:54:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:54:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:54:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_4 2023-12-02T22:54:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2023-12-02T22:54:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-02T22:54:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-02T22:54:42.252 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6302571251711659) > weight (0.25) 2023-12-02T22:54:42.252 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.674730302110042) > weight (0.25) 2023-12-02T22:54:42.252 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.42725667914633203) > weight (0.25) 2023-12-02T22:54:42.252 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5978267132867701) > weight (0.25) 2023-12-02T22:54:42.252 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-02T22:54:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-02T22:54:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-02T22:54:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:54:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:54:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-02T22:54:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:54:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-02T22:54:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_5': 2023-12-02T22:54:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:54:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:54:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:54:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 193fb099fd24 2023-12-02T22:54:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.193fb099fd24 2023-12-02T22:54:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:54:42 2023 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:54:42 2023 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:54:42 2023 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:54:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 2d5ec3af-0b39-4b21-af38-80e1ccade13a 2023-12-02T22:54:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:54:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-02T22:54:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:54:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:54:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2023-12-02T22:54:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:54:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 5444608 bytes 2023-12-02T22:54:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 90300416 bytes 2023-12-02T22:54:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 223659008 bytes 2023-12-02T22:54:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 240443392 bytes 2023-12-02T22:54:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 306510848 bytes 2023-12-02T22:54:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 312143872 bytes 2023-12-02T22:54:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 353479680 bytes 2023-12-02T22:54:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 400926720 bytes 2023-12-02T22:54:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 442383360 bytes 2023-12-02T22:54:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 477947904 bytes 2023-12-02T22:54:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 527031296 bytes 2023-12-02T22:54:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 559628288 bytes 2023-12-02T22:54:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 612442112 bytes 2023-12-02T22:54:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 649037824 bytes 2023-12-02T22:54:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 676533248 bytes 2023-12-02T22:54:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 745109504 bytes 2023-12-02T22:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 770608128 bytes 2023-12-02T22:54:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 858412032 bytes 2023-12-02T22:54:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 963981312 bytes 2023-12-02T22:54:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 987953152 bytes 2023-12-02T22:54:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 988542976 bytes 2023-12-02T22:54:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1017242624 bytes 2023-12-02T22:54:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1039800320 bytes 2023-12-02T22:54:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1066374144 bytes 2023-12-02T22:54:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1152482304 bytes 2023-12-02T22:54:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1207120896 bytes 2023-12-02T22:54:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1288381440 bytes 2023-12-02T22:54:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1317670912 bytes 2023-12-02T22:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1356311552 bytes 2023-12-02T22:54:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1410725888 bytes 2023-12-02T22:54:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1447087104 bytes 2023-12-02T22:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1464083456 bytes 2023-12-02T22:54:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1527934976 bytes 2023-12-02T22:54:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1567369216 bytes 2023-12-02T22:54:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1594855424 bytes 2023-12-02T22:54:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1665251328 bytes 2023-12-02T22:54:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1673621504 bytes 2023-12-02T22:54:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1698601984 bytes 2023-12-02T22:54:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1715387392 bytes 2023-12-02T22:54:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1764047872 bytes 2023-12-02T22:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1807442944 bytes 2023-12-02T22:54:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1833213952 bytes 2023-12-02T22:54:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1861498880 bytes 2023-12-02T22:54:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1862103040 bytes 2023-12-02T22:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1881585664 bytes 2023-12-02T22:54:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1893862400 bytes 2023-12-02T22:54:49.344 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4475457860855897) > weight (0.25) 2023-12-02T22:54:49.344 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4153383797929976) > weight (0.25) 2023-12-02T22:54:49.344 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.836625306879933) > weight (0.25) 2023-12-02T22:54:49.345 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4067954740192904) > weight (0.25) 2023-12-02T22:54:49.345 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-02T22:54:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1955954688 bytes 2023-12-02T22:54:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2025544704 bytes 2023-12-02T22:54:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2060924928 bytes 2023-12-02T22:54:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2107825152 bytes 2023-12-02T22:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2164280320 bytes 2023-12-02T22:54:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2254761984 bytes 2023-12-02T22:54:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2355292160 bytes 2023-12-02T22:54:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2426786816 bytes 2023-12-02T22:54:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2489740288 bytes 2023-12-02T22:54:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2560839680 bytes 2023-12-02T22:54:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2653912064 bytes 2023-12-02T22:54:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2702384128 bytes 2023-12-02T22:54:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2750287872 bytes 2023-12-02T22:54:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2769697792 bytes 2023-12-02T22:54:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2836641792 bytes 2023-12-02T22:54:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2906118144 bytes 2023-12-02T22:54:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2921755648 bytes 2023-12-02T22:54:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2961932288 bytes 2023-12-02T22:54:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 3019853824 bytes 2023-12-02T22:54:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 3051033600 bytes 2023-12-02T22:54:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3059025920 bytes 2023-12-02T22:54:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3113443328 bytes 2023-12-02T22:54:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3165814784 bytes 2023-12-02T22:54:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3192495104 bytes 2023-12-02T22:54:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3226419200 bytes 2023-12-02T22:54:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3234613248 bytes 2023-12-02T22:54:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3267478528 bytes 2023-12-02T22:54:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3283076096 bytes 2023-12-02T22:54:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3340206080 bytes 2023-12-02T22:54:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3426292736 bytes 2023-12-02T22:54:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3455108096 bytes 2023-12-02T22:54:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3461369856 bytes 2023-12-02T22:54:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3504787456 bytes 2023-12-02T22:54:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3516673024 bytes 2023-12-02T22:54:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3600185344 bytes 2023-12-02T22:54:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3644320768 bytes 2023-12-02T22:54:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3658667008 bytes 2023-12-02T22:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3686850560 bytes 2023-12-02T22:54:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3732368384 bytes 2023-12-02T22:54:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3772144640 bytes 2023-12-02T22:54:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3789817856 bytes 2023-12-02T22:54:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3883366400 bytes 2023-12-02T22:54:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3924588544 bytes 2023-12-02T22:54:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 4012250112 bytes 2023-12-02T22:54:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 4057540608 bytes 2023-12-02T22:54:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:54:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:54:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:54:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:54:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-02T22:54:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-02T22:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:54:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=15f84b78131e 2023-12-02T22:54:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:54:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_5 present 15f84b78131e 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=15f84b78131e 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2023-12-02T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:54:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:54:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2023-12-02T22:54:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-02T22:54:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:54:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=15f84b78131e 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 15f84b78131e 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 15f84b78131e -o 15f84b78131e = 15f84b78131e 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:54:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-02T22:55:10.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:55:10.450+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff15927520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:55:10.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:55:10.450+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff1531e000 [3/2d5ec3af-0b39-4b21-af38-80e1ccade13a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:55:27.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:55:27.271+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fca07400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-02T22:55:27.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:55:27.271+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fca07400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-02T22:55:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:55:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:55:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2023-12-02T22:55:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:55:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:55:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2023-12-02T22:55:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-02T22:55:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:55:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-02T22:55:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:55:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:55:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2023-12-02T22:55:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:55:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:55:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2023-12-02T22:55:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-02T22:55:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:55:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-02T22:55:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:55:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-02T22:55:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-02T22:55:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:55:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:55:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:55:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_5 2023-12-02T22:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2023-12-02T22:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-02T22:55:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-02T22:55:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:55:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:55:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-02T22:55:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-02T22:55:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:55:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:55:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-02T22:55:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:55:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-02T22:55:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_6': 2023-12-02T22:55:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1957c53c42f7 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1957c53c42f7 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:55:28 2023 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:55:28 2023 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:55:28 2023 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:55:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:55:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 6acbf29b-4df6-43f0-94ad-2bb4641ae153 2023-12-02T22:55:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:55:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-02T22:55:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:55:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:55:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2023-12-02T22:55:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:55:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:55:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 22075392 bytes 2023-12-02T22:55:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 32117760 bytes 2023-12-02T22:55:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 46867456 bytes 2023-12-02T22:55:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 62640128 bytes 2023-12-02T22:55:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 130731008 bytes 2023-12-02T22:55:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 159584256 bytes 2023-12-02T22:55:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 202999808 bytes 2023-12-02T22:55:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 255184896 bytes 2023-12-02T22:55:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 265324544 bytes 2023-12-02T22:55:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 329457664 bytes 2023-12-02T22:55:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 378461184 bytes 2023-12-02T22:55:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 492126208 bytes 2023-12-02T22:55:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 558389248 bytes 2023-12-02T22:55:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 590987264 bytes 2023-12-02T22:55:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 629246976 bytes 2023-12-02T22:55:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 654179328 bytes 2023-12-02T22:55:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 694706176 bytes 2023-12-02T22:55:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 793736192 bytes 2023-12-02T22:55:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 814123008 bytes 2023-12-02T22:55:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 838875136 bytes 2023-12-02T22:55:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 917442560 bytes 2023-12-02T22:55:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 944422912 bytes 2023-12-02T22:55:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1023689728 bytes 2023-12-02T22:55:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1053374464 bytes 2023-12-02T22:55:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1127298048 bytes 2023-12-02T22:55:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1145098240 bytes 2023-12-02T22:55:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1195632640 bytes 2023-12-02T22:55:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1252127744 bytes 2023-12-02T22:55:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1310159872 bytes 2023-12-02T22:55:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1374764032 bytes 2023-12-02T22:55:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1404260352 bytes 2023-12-02T22:55:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1452662784 bytes 2023-12-02T22:55:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1490356224 bytes 2023-12-02T22:55:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1511776256 bytes 2023-12-02T22:55:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1606248448 bytes 2023-12-02T22:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1660188672 bytes 2023-12-02T22:55:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1689932800 bytes 2023-12-02T22:55:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1704654848 bytes 2023-12-02T22:55:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1776727040 bytes 2023-12-02T22:55:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1797380096 bytes 2023-12-02T22:55:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1816175616 bytes 2023-12-02T22:55:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1882433536 bytes 2023-12-02T22:55:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1935734784 bytes 2023-12-02T22:55:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1989962752 bytes 2023-12-02T22:55:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2020825088 bytes 2023-12-02T22:55:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2081176576 bytes 2023-12-02T22:55:34.978 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:55:34.976+0000 7f315ad82640 -1 reset not still connected to 0x564dee1c6a90 2023-12-02T22:55:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2103464960 bytes 2023-12-02T22:55:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2162415616 bytes 2023-12-02T22:55:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2230073344 bytes 2023-12-02T22:55:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2241612800 bytes 2023-12-02T22:55:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2254398464 bytes 2023-12-02T22:55:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2281276416 bytes 2023-12-02T22:55:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2321577984 bytes 2023-12-02T22:55:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2353574912 bytes 2023-12-02T22:55:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2410663936 bytes 2023-12-02T22:55:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2469904384 bytes 2023-12-02T22:55:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2563819520 bytes 2023-12-02T22:55:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2612439040 bytes 2023-12-02T22:55:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2629565440 bytes 2023-12-02T22:55:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2729529344 bytes 2023-12-02T22:55:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2803026944 bytes 2023-12-02T22:55:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2863917056 bytes 2023-12-02T22:55:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2901009408 bytes 2023-12-02T22:55:37.390 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:55:37.389+0000 7f315f151640 -1 bdev(0x564dd37a4e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-02T22:55:37.452 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f315f151640 time 2023-12-02T22:55:37.451715+0000 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x564dd00e7e3b] 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:*** Caught signal (Aborted) ** 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: in thread 7f315f151640 thread_name:bstore_aio 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:55:37.451+0000 7f315f151640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f315f151640 time 2023-12-02T22:55:37.451715+0000 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.453 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x564dd00e7e3b] 2023-12-02T22:55:37.454 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.454 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.454 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.454 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.454 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.454 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f316c854db0] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f316c8a154c] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: raise() 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: abort() 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x564dd00e7efc] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 7: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2023-12-02T22:55:37.452+0000 7f315f151640 -1 *** Caught signal (Aborted) ** 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: in thread 7f315f151640 thread_name:bstore_aio 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f316c854db0] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f316c8a154c] 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: raise() 2023-12-02T22:55:37.455 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: abort() 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x564dd00e7efc] 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 7: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T22:55:37.456 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.462 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: -536> 2023-12-02T22:55:37.389+0000 7f315f151640 -1 bdev(0x564dd37a4e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: -1> 2023-12-02T22:55:37.451+0000 7f315f151640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f315f151640 time 2023-12-02T22:55:37.451715+0000 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x564dd00e7e3b] 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 0> 2023-12-02T22:55:37.452+0000 7f315f151640 -1 *** Caught signal (Aborted) ** 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: in thread 7f315f151640 thread_name:bstore_aio 2023-12-02T22:55:37.463 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f316c854db0] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f316c8a154c] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: raise() 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: abort() 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x564dd00e7efc] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 7: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T22:55:37.464 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.471 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: -536> 2023-12-02T22:55:37.389+0000 7f315f151640 -1 bdev(0x564dd37a4e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: -1> 2023-12-02T22:55:37.451+0000 7f315f151640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f315f151640 time 2023-12-02T22:55:37.451715+0000 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7670-g0c523c70/rpm/el9/BUILD/ceph-18.0.0-7670-g0c523c70/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x564dd00e7e3b] 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.472 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 0> 2023-12-02T22:55:37.452+0000 7f315f151640 -1 *** Caught signal (Aborted) ** 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: in thread 7f315f151640 thread_name:bstore_aio 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev) 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f316c854db0] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f316c8a154c] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 3: raise() 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 4: abort() 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x564dd00e7efc] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x564dd0a36da6] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 7: ceph-osd(+0xd9ff91) [0x564dd0a17f91] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f316c89f802] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f316c83f450] 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T22:55:37.473 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr: 2023-12-02T22:55:37.625 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:daemon-helper: command failed with exit status 5 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:55:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-02T22:55:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:55:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12c2fa52c630 2023-12-02T22:55:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:55:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_6 present 12c2fa52c630 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c2fa52c630 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2023-12-02T22:55:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:55:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:55:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2023-12-02T22:55:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-02T22:55:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:55:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c2fa52c630 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c2fa52c630 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c2fa52c630 -o 12c2fa52c630 = 12c2fa52c630 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-02T22:55:39.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-02T22:55:45.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-02T22:55:50.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:55:50.553+0000 7f6bd6d28640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563249c4b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:55:50.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:55:50.553+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a54a500 [3/6acbf29b-4df6-43f0-94ad-2bb4641ae153] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:55:50.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-02T22:55:56.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-02T22:56:01.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-02T22:56:06.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-02T22:56:10.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:56:10.892+0000 7fd85d935640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e335b99800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-02T22:56:10.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:56:10.892+0000 7fd85d935640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e335b99800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-12-02T22:56:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:56:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-02T22:56:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:56:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:56:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-02T22:56:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-02T22:56:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:56:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:56:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:56:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-02T22:56:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:56:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-02T22:56:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-02T22:56:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:56:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:56:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:56:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_6 2023-12-02T22:56:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2023-12-02T22:56:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-02T22:56:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-02T22:56:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:56:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:56:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:56:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2023-12-02T22:56:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-02T22:56:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-02T22:56:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-02T22:56:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:56:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-02T22:56:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:56:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-02T22:56:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_7': 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 196f784c7661 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.196f784c7661 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:56:11 2023 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:56:11 2023 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:56:11 2023 2023-12-02T22:56:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:56:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:56:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d5952b16-2cb9-4973-9317-95871c33971d 2023-12-02T22:56:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:56:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-02T22:56:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:56:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2023-12-02T22:56:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:56:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:56:12.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-02T22:56:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 80936960 bytes 2023-12-02T22:56:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 113048576 bytes 2023-12-02T22:56:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 162114560 bytes 2023-12-02T22:56:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 177745920 bytes 2023-12-02T22:56:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 237477888 bytes 2023-12-02T22:56:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 289380352 bytes 2023-12-02T22:56:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 349218816 bytes 2023-12-02T22:56:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 403437568 bytes 2023-12-02T22:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 439915520 bytes 2023-12-02T22:56:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 557519872 bytes 2023-12-02T22:56:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 558443520 bytes 2023-12-02T22:56:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 616546304 bytes 2023-12-02T22:56:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 640791552 bytes 2023-12-02T22:56:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 707938304 bytes 2023-12-02T22:56:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 728303616 bytes 2023-12-02T22:56:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 794515456 bytes 2023-12-02T22:56:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 875206656 bytes 2023-12-02T22:56:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 960416768 bytes 2023-12-02T22:56:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 1023384576 bytes 2023-12-02T22:56:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 1060960256 bytes 2023-12-02T22:56:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 1078554624 bytes 2023-12-02T22:56:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1148139520 bytes 2023-12-02T22:56:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1205570560 bytes 2023-12-02T22:56:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1209424896 bytes 2023-12-02T22:56:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1247842304 bytes 2023-12-02T22:56:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1248459776 bytes 2023-12-02T22:56:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1276448768 bytes 2023-12-02T22:56:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1308256256 bytes 2023-12-02T22:56:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1335682048 bytes 2023-12-02T22:56:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1393256448 bytes 2023-12-02T22:56:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1425722368 bytes 2023-12-02T22:56:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1496397824 bytes 2023-12-02T22:56:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1507254272 bytes 2023-12-02T22:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1608196096 bytes 2023-12-02T22:56:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1665105920 bytes 2023-12-02T22:56:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1685469184 bytes 2023-12-02T22:56:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1746022400 bytes 2023-12-02T22:56:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1757047808 bytes 2023-12-02T22:56:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1803674624 bytes 2023-12-02T22:56:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1828057088 bytes 2023-12-02T22:56:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1873383424 bytes 2023-12-02T22:56:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1912647680 bytes 2023-12-02T22:56:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1938168832 bytes 2023-12-02T22:56:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1960747008 bytes 2023-12-02T22:56:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1961364480 bytes 2023-12-02T22:56:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1963978752 bytes 2023-12-02T22:56:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2014423040 bytes 2023-12-02T22:56:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2058646528 bytes 2023-12-02T22:56:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2059251712 bytes 2023-12-02T22:56:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2086395904 bytes 2023-12-02T22:56:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2092407808 bytes 2023-12-02T22:56:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2142270464 bytes 2023-12-02T22:56:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2213280768 bytes 2023-12-02T22:56:17.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-02T22:56:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2282832896 bytes 2023-12-02T22:56:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2343690240 bytes 2023-12-02T22:56:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2376515584 bytes 2023-12-02T22:56:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2384063488 bytes 2023-12-02T22:56:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2435706880 bytes 2023-12-02T22:56:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2482674688 bytes 2023-12-02T22:56:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2513943552 bytes 2023-12-02T22:56:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2574284800 bytes 2023-12-02T22:56:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2642416640 bytes 2023-12-02T22:56:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2676476928 bytes 2023-12-02T22:56:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2757179392 bytes 2023-12-02T22:56:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2790237184 bytes 2023-12-02T22:56:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2821213184 bytes 2023-12-02T22:56:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2873818112 bytes 2023-12-02T22:56:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:56:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12c2906066e7 2023-12-02T22:56:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:56:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:56:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 present 12c2906066e7 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c2906066e7 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c2906066e7 2023-12-02T22:56:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c2906066e7 2023-12-02T22:56:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c2906066e7 -o 12c2906066e7 = 12c2906066e7 2023-12-02T22:56:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-02T22:56:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:56:20.553+0000 7f6bd7529640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56324ab23a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:56:20.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:56:20.553+0000 7f6bddd36640 -1 rbd::mirror::ImageReplayer: 0x56324a698780 [3/d5952b16-2cb9-4973-9317-95871c33971d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:56:23.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-02T22:56:23.253 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5089516145032126) > weight (0.25) 2023-12-02T22:56:23.253 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7158043961857106) > weight (0.25) 2023-12-02T22:56:23.253 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5104771721498658) > weight (0.25) 2023-12-02T22:56:23.253 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T22:56:23.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T22:56:23.254 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:56:23.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:56:23.254+0000 7fd86b150640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 103556) UID: 1000 2023-12-02T22:56:28.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-02T22:56:30.346 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T22:56:30.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T22:56:30.346 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-02T22:56:30.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:56:30.347+0000 7f6bde537640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 102596) UID: 1000 2023-12-02T22:56:32.254 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T22:56:32.255 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:56:32.255 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T22:56:32.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T22:56:32.255 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T22:56:32.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T22:56:32.258 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-02T22:56:33.998 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-02T22:56:38.347 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T22:56:38.347 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:56:38.348 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T22:56:38.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:56:38.348 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:56:38.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:56:38.351 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-02T22:56:39.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-02T22:56:44.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-02T22:56:49.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:56:49.915+0000 7fec6f335640 -1 rbd::mirror::ImageMap: 0x5583f83d6c60 handle_peer_ack_remove: failed to remove global_image_id=d5952b16-2cb9-4973-9317-95871c33971d 2023-12-02T22:56:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=12c2906066e7 2023-12-02T22:56:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:56:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:56:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted 12c2906066e7 2023-12-02T22:56:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c2906066e7 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c2906066e7 2023-12-02T22:56:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c2906066e7 2023-12-02T22:56:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c2906066e7 -o 12c2906066e7 = 12c2906066e7 2023-12-02T22:56:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:56:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:56:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:56:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:56:50.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-02T22:56:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted 12c2906066e7 2023-12-02T22:56:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:56:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c2906066e7 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c2906066e7 2023-12-02T22:56:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c2906066e7 2023-12-02T22:56:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c2906066e7 -o 12c2906066e7 = 12c2906066e7 2023-12-02T22:56:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:56:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:56:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:56:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted 12c2906066e7 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c2906066e7 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:56:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=12c2906066e7 2023-12-02T22:56:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 12c2906066e7 2023-12-02T22:56:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 12c2906066e7 -o 12c2906066e7 = 12c2906066e7 2023-12-02T22:56:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:56:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = present 2023-12-02T22:56:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:56:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 4 2023-12-02T22:56:55.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-02T22:56:56.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:56:56.392+0000 7f20dfb2c640 -1 rbd::mirror::ImageMap: 0x55ff14facea0 handle_peer_ack_remove: failed to remove global_image_id=d5952b16-2cb9-4973-9317-95871c33971d 2023-12-02T22:56:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted 12c2906066e7 2023-12-02T22:56:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:56:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=12c2906066e7 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:56:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2023-12-02T22:56:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-02T22:56:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:56:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-02T22:56:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-02T22:56:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:56:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:56:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_7 2023-12-02T22:56:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2023-12-02T22:56:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-02T22:56:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-02T22:56:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:56:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:56:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-02T22:56:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-02T22:56:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:56:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:56:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-02T22:56:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:56:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-02T22:56:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_8': 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 19974a70bff6 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.19974a70bff6 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:56:57 2023 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:56:57 2023 2023-12-02T22:56:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:56:57 2023 2023-12-02T22:56:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:56:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:56:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f92e0af5-c281-4b1d-aed1-bf7042a787c8 2023-12-02T22:56:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:56:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-02T22:56:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:56:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:56:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:56:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:56:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:56:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:56:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:56:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 61948928 bytes 2023-12-02T22:56:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 81977344 bytes 2023-12-02T22:56:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 110249984 bytes 2023-12-02T22:56:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 128536576 bytes 2023-12-02T22:56:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 156390400 bytes 2023-12-02T22:56:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 232529920 bytes 2023-12-02T22:56:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 275355648 bytes 2023-12-02T22:56:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 275988480 bytes 2023-12-02T22:56:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 296733696 bytes 2023-12-02T22:56:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 370608128 bytes 2023-12-02T22:56:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 391666688 bytes 2023-12-02T22:56:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 445080576 bytes 2023-12-02T22:56:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 506360832 bytes 2023-12-02T22:56:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 514949120 bytes 2023-12-02T22:56:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 539760640 bytes 2023-12-02T22:56:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 540355584 bytes 2023-12-02T22:56:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 564497408 bytes 2023-12-02T22:56:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 565129216 bytes 2023-12-02T22:56:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 635316224 bytes 2023-12-02T22:56:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 635964416 bytes 2023-12-02T22:56:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 637632512 bytes 2023-12-02T22:56:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 708442112 bytes 2023-12-02T22:57:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 722814976 bytes 2023-12-02T22:57:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 768759808 bytes 2023-12-02T22:57:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 792651776 bytes 2023-12-02T22:57:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 840929280 bytes 2023-12-02T22:57:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 900716544 bytes 2023-12-02T22:57:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 946443264 bytes 2023-12-02T22:57:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 951691264 bytes 2023-12-02T22:57:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1040137216 bytes 2023-12-02T22:57:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1046119424 bytes 2023-12-02T22:57:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1056677888 bytes 2023-12-02T22:57:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1140166656 bytes 2023-12-02T22:57:01.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-02T22:57:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1213353984 bytes 2023-12-02T22:57:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1233597440 bytes 2023-12-02T22:57:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1273717760 bytes 2023-12-02T22:57:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1290502144 bytes 2023-12-02T22:57:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1336327168 bytes 2023-12-02T22:57:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1368040448 bytes 2023-12-02T22:57:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1449481216 bytes 2023-12-02T22:57:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1506405376 bytes 2023-12-02T22:57:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1524377600 bytes 2023-12-02T22:57:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1534708736 bytes 2023-12-02T22:57:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1608568832 bytes 2023-12-02T22:57:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1666099200 bytes 2023-12-02T22:57:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1756426240 bytes 2023-12-02T22:57:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1800691712 bytes 2023-12-02T22:57:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1844822016 bytes 2023-12-02T22:57:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1904308224 bytes 2023-12-02T22:57:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1959899136 bytes 2023-12-02T22:57:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2027070464 bytes 2023-12-02T22:57:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2046603264 bytes 2023-12-02T22:57:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2094275584 bytes 2023-12-02T22:57:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2110387200 bytes 2023-12-02T22:57:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2138310656 bytes 2023-12-02T22:57:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2147664896 bytes 2023-12-02T22:57:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2203590656 bytes 2023-12-02T22:57:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2242161664 bytes 2023-12-02T22:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2279836672 bytes 2023-12-02T22:57:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2331753472 bytes 2023-12-02T22:57:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2395487232 bytes 2023-12-02T22:57:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2403520512 bytes 2023-12-02T22:57:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2444686336 bytes 2023-12-02T22:57:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2516204544 bytes 2023-12-02T22:57:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2548881408 bytes 2023-12-02T22:57:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2594321408 bytes 2023-12-02T22:57:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2634027008 bytes 2023-12-02T22:57:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2715429888 bytes 2023-12-02T22:57:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2774007808 bytes 2023-12-02T22:57:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2810720256 bytes 2023-12-02T22:57:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2824475648 bytes 2023-12-02T22:57:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2849537024 bytes 2023-12-02T22:57:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2889388032 bytes 2023-12-02T22:57:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2936115200 bytes 2023-12-02T22:57:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 2977264640 bytes 2023-12-02T22:57:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 2991588352 bytes 2023-12-02T22:57:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3024347136 bytes 2023-12-02T22:57:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3078519808 bytes 2023-12-02T22:57:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3132294144 bytes 2023-12-02T22:57:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3143149568 bytes 2023-12-02T22:57:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3163275264 bytes 2023-12-02T22:57:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3223847936 bytes 2023-12-02T22:57:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3240675328 bytes 2023-12-02T22:57:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3298049024 bytes 2023-12-02T22:57:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3315209216 bytes 2023-12-02T22:57:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3375457280 bytes 2023-12-02T22:57:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3376065536 bytes 2023-12-02T22:57:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3410320384 bytes 2023-12-02T22:57:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3449722880 bytes 2023-12-02T22:57:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3501792256 bytes 2023-12-02T22:57:06.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-02T22:57:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3530212352 bytes 2023-12-02T22:57:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3641508864 bytes 2023-12-02T22:57:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3689414656 bytes 2023-12-02T22:57:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 3748273152 bytes 2023-12-02T22:57:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 3763605504 bytes 2023-12-02T22:57:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 3798028288 bytes 2023-12-02T22:57:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 3835607040 bytes 2023-12-02T22:57:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 3886501888 bytes 2023-12-02T22:57:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 3938732032 bytes 2023-12-02T22:57:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 3981894656 bytes 2023-12-02T22:57:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4029199360 bytes 2023-12-02T22:57:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4060804096 bytes 2023-12-02T22:57:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4115333120 bytes 2023-12-02T22:57:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4133554176 bytes 2023-12-02T22:57:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4142501888 bytes 2023-12-02T22:57:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4180543488 bytes 2023-12-02T22:57:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4231022592 bytes 2023-12-02T22:57:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10800 IOs, 4294060032 bytes 2023-12-02T22:57:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10900 IOs, 4335976448 bytes 2023-12-02T22:57:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11000 IOs, 4387391488 bytes 2023-12-02T22:57:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11100 IOs, 4504998912 bytes 2023-12-02T22:57:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11200 IOs, 4586904576 bytes 2023-12-02T22:57:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11300 IOs, 4661985280 bytes 2023-12-02T22:57:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11400 IOs, 4728182784 bytes 2023-12-02T22:57:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11500 IOs, 4760715264 bytes 2023-12-02T22:57:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11600 IOs, 4791896064 bytes 2023-12-02T22:57:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11700 IOs, 4884977664 bytes 2023-12-02T22:57:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11800 IOs, 4932939776 bytes 2023-12-02T22:57:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11900 IOs, 4933529600 bytes 2023-12-02T22:57:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12000 IOs, 4952966144 bytes 2023-12-02T22:57:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12100 IOs, 4966352896 bytes 2023-12-02T22:57:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12200 IOs, 5005282304 bytes 2023-12-02T22:57:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12300 IOs, 5005881344 bytes 2023-12-02T22:57:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:57:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2023-12-02T22:57:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:57:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:57:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2023-12-02T22:57:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-02T22:57:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=15f830ce0e2c 2023-12-02T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:57:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_8 present 15f830ce0e2c 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=15f830ce0e2c 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2023-12-02T22:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2023-12-02T22:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-02T22:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:57:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=15f830ce0e2c 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 15f830ce0e2c 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 15f830ce0e2c -o 15f830ce0e2c = 15f830ce0e2c 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-02T22:57:11.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-02T22:57:17.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-02T22:57:22.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-02T22:57:26.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:57:26.480+0000 7faf1fcd4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee40394ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:57:26.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:57:26.480+0000 7faf26ce2640 -1 rbd::mirror::ImageReplayer: 0x55ee40434c80 [3/f92e0af5-c281-4b1d-aed1-bf7042a787c8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:57:28.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-02T22:57:33.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-02T22:57:38.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-02T22:57:39.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:57:39.929+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fca2e000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-02T22:57:39.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:57:39.929+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fca2e000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-02T22:57:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:57:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2023-12-02T22:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-02T22:57:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:57:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-02T22:57:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:57:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:57:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:57:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-02T22:57:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:57:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-02T22:57:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:57:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_8 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-12-02T22:57:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-02T22:57:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:57:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:57:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_8 2023-12-02T22:57:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2023-12-02T22:57:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-02T22:57:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-02T22:57:40.352 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T22:57:40.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T22:57:40.353 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-02T22:57:40.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:57:40.352+0000 7f6d14800640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 125590) UID: 1000 2023-12-02T22:57:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:57:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:57:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:57:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2023-12-02T22:57:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_9 512M 2023-12-02T22:57:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:57:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-02T22:57:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_9 2023-12-02T22:57:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:57:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:57:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-02T22:57:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:57:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-02T22:57:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_9': 2023-12-02T22:57:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:57:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:57:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 19af733e2475 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.19af733e2475 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:57:40 2023 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:57:40 2023 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:57:40 2023 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 31728b5c-e341-4eb2-b849-bc62db59e906 2023-12-02T22:57:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:57:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-02T22:57:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:57:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:57:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2023-12-02T22:57:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:57:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:57:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 43284480 bytes 2023-12-02T22:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 74868736 bytes 2023-12-02T22:57:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 95785984 bytes 2023-12-02T22:57:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 125670400 bytes 2023-12-02T22:57:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 163481600 bytes 2023-12-02T22:57:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 188148736 bytes 2023-12-02T22:57:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 225152000 bytes 2023-12-02T22:57:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 248957952 bytes 2023-12-02T22:57:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 305017856 bytes 2023-12-02T22:57:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 328251392 bytes 2023-12-02T22:57:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 342747136 bytes 2023-12-02T22:57:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 362894336 bytes 2023-12-02T22:57:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 383653888 bytes 2023-12-02T22:57:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 397156352 bytes 2023-12-02T22:57:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 428073984 bytes 2023-12-02T22:57:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 467884032 bytes 2023-12-02T22:57:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 480952320 bytes 2023-12-02T22:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 545258496 bytes 2023-12-02T22:57:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 592580608 bytes 2023-12-02T22:57:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 593193984 bytes 2023-12-02T22:57:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 618739712 bytes 2023-12-02T22:57:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 624724992 bytes 2023-12-02T22:57:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 642869248 bytes 2023-12-02T22:57:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 688946176 bytes 2023-12-02T22:57:43.354 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T22:57:43.354 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:57:43.354 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T22:57:43.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T22:57:43.354 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T22:57:43.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T22:57:43.357 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-02T22:57:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 707771392 bytes 2023-12-02T22:57:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 754283520 bytes 2023-12-02T22:57:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 792324096 bytes 2023-12-02T22:57:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 842475520 bytes 2023-12-02T22:57:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 866760704 bytes 2023-12-02T22:57:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 877286400 bytes 2023-12-02T22:57:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 907087872 bytes 2023-12-02T22:57:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 930714624 bytes 2023-12-02T22:57:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 984059904 bytes 2023-12-02T22:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1017843712 bytes 2023-12-02T22:57:44.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-02T22:57:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1038449664 bytes 2023-12-02T22:57:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1057831936 bytes 2023-12-02T22:57:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1105913856 bytes 2023-12-02T22:57:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1190813696 bytes 2023-12-02T22:57:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1235497984 bytes 2023-12-02T22:57:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1305807872 bytes 2023-12-02T22:57:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1347606528 bytes 2023-12-02T22:57:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1379122176 bytes 2023-12-02T22:57:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 1417237504 bytes 2023-12-02T22:57:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 1443521536 bytes 2023-12-02T22:57:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 1449263104 bytes 2023-12-02T22:57:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 1527455744 bytes 2023-12-02T22:57:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 1540970496 bytes 2023-12-02T22:57:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 1596323840 bytes 2023-12-02T22:57:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 1656190976 bytes 2023-12-02T22:57:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 1697199104 bytes 2023-12-02T22:57:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 1766350848 bytes 2023-12-02T22:57:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 1872906240 bytes 2023-12-02T22:57:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 1889566720 bytes 2023-12-02T22:57:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 1925326848 bytes 2023-12-02T22:57:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 1970585600 bytes 2023-12-02T22:57:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2015840256 bytes 2023-12-02T22:57:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2040624128 bytes 2023-12-02T22:57:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2068001792 bytes 2023-12-02T22:57:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2117258240 bytes 2023-12-02T22:57:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2170774528 bytes 2023-12-02T22:57:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2196169728 bytes 2023-12-02T22:57:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2222679040 bytes 2023-12-02T22:57:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2242426880 bytes 2023-12-02T22:57:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2293456896 bytes 2023-12-02T22:57:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2327294976 bytes 2023-12-02T22:57:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 2368873472 bytes 2023-12-02T22:57:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 2441935872 bytes 2023-12-02T22:57:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 2442549248 bytes 2023-12-02T22:57:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 2518522880 bytes 2023-12-02T22:57:47.496 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T22:57:47.494+0000 7f4a27c08640 -1 reset not still connected to 0x55cd826dba00 2023-12-02T22:57:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 2519109632 bytes 2023-12-02T22:57:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 2545159168 bytes 2023-12-02T22:57:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 2559566848 bytes 2023-12-02T22:57:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 2560171008 bytes 2023-12-02T22:57:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 2669103104 bytes 2023-12-02T22:57:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 2722319360 bytes 2023-12-02T22:57:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 2785402880 bytes 2023-12-02T22:57:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 2836813824 bytes 2023-12-02T22:57:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 2903436288 bytes 2023-12-02T22:57:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 2960007168 bytes 2023-12-02T22:57:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 2994472960 bytes 2023-12-02T22:57:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3022599168 bytes 2023-12-02T22:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3024760832 bytes 2023-12-02T22:57:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3102259200 bytes 2023-12-02T22:57:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3102874624 bytes 2023-12-02T22:57:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3141100544 bytes 2023-12-02T22:57:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3206056960 bytes 2023-12-02T22:57:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3238223872 bytes 2023-12-02T22:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3271138304 bytes 2023-12-02T22:57:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3359787008 bytes 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:57:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:57:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:57:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2023-12-02T22:57:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:57:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:57:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2023-12-02T22:57:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-02T22:57:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:57:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=15f8bdd1da93 2023-12-02T22:57:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:57:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_9 present 15f8bdd1da93 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=15f8bdd1da93 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:57:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:57:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2023-12-02T22:57:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:57:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:57:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2023-12-02T22:57:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-02T22:57:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:57:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=15f8bdd1da93 2023-12-02T22:57:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 15f8bdd1da93 2023-12-02T22:57:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 15f8bdd1da93 -o 15f8bdd1da93 = 15f8bdd1da93 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_9 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:57:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-12-02T22:57:49.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-02T22:57:55.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-02T22:58:00.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~140s 2023-12-02T22:58:01.259 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3799436823252049) > weight (0.25) 2023-12-02T22:58:01.259 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.266815488751458) > weight (0.25) 2023-12-02T22:58:01.259 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4963072706817361) > weight (0.25) 2023-12-02T22:58:01.259 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T22:58:01.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T22:58:01.260 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T22:58:01.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:58:01.260+0000 7f92e1d25640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 125484) UID: 1000 2023-12-02T22:58:05.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-02T22:58:10.261 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T22:58:10.261 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:58:10.262 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T22:58:10.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T22:58:10.262 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T22:58:10.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T22:58:10.265 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-02T22:58:10.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:58:10.450+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff15a4e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:58:10.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:58:10.450+0000 7f20dfb2c640 -1 rbd::mirror::ImageReplayer: 0x55ff1531e000 [3/31728b5c-e341-4eb2-b849-bc62db59e906] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:58:11.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-02T22:58:16.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-02T22:58:20.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:58:20.944+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fbe17000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-12-02T22:58:20.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:58:20.944+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fbe17000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-12-02T22:58:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:58:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-02T22:58:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:58:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-02T22:58:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:58:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:58:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:58:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-02T22:58:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:58:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-02T22:58:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:58:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_9 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-12-02T22:58:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-02T22:58:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:58:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:58:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:58:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_9 2023-12-02T22:58:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2023-12-02T22:58:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-02T22:58:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-02T22:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_10 512M 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2023-12-02T22:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-02T22:58:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_10 2023-12-02T22:58:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:58:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=snapshot 2023-12-02T22:58:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-02T22:58:22.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-02T22:58:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2023-12-02T22:58:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_10': 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 19d7de728587 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.19d7de728587 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Dec 2 22:58:21 2023 2023-12-02T22:58:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Dec 2 22:58:21 2023 2023-12-02T22:58:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Dec 2 22:58:21 2023 2023-12-02T22:58:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2023-12-02T22:58:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: snapshot 2023-12-02T22:58:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 1d37770c-44a0-4edf-937b-cf42e3d672b1 2023-12-02T22:58:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2023-12-02T22:58:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-02T22:58:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:58:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T22:58:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2023-12-02T22:58:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:58:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-02T22:58:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 81765376 bytes 2023-12-02T22:58:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 154601472 bytes 2023-12-02T22:58:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 192348160 bytes 2023-12-02T22:58:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 226451456 bytes 2023-12-02T22:58:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 287152128 bytes 2023-12-02T22:58:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 303705088 bytes 2023-12-02T22:58:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 380045312 bytes 2023-12-02T22:58:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 449112064 bytes 2023-12-02T22:58:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 457090048 bytes 2023-12-02T22:58:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1000 IOs, 520858624 bytes 2023-12-02T22:58:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1100 IOs, 539179008 bytes 2023-12-02T22:58:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1200 IOs, 579938304 bytes 2023-12-02T22:58:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1300 IOs, 588727296 bytes 2023-12-02T22:58:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1400 IOs, 611012608 bytes 2023-12-02T22:58:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1500 IOs, 627670016 bytes 2023-12-02T22:58:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1600 IOs, 667085824 bytes 2023-12-02T22:58:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1700 IOs, 702205952 bytes 2023-12-02T22:58:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1800 IOs, 779939840 bytes 2023-12-02T22:58:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:1900 IOs, 835103744 bytes 2023-12-02T22:58:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2000 IOs, 940980224 bytes 2023-12-02T22:58:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2100 IOs, 969134080 bytes 2023-12-02T22:58:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2200 IOs, 1060235264 bytes 2023-12-02T22:58:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2300 IOs, 1060841472 bytes 2023-12-02T22:58:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2400 IOs, 1123732480 bytes 2023-12-02T22:58:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2500 IOs, 1180708864 bytes 2023-12-02T22:58:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2600 IOs, 1207454720 bytes 2023-12-02T22:58:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2700 IOs, 1293521920 bytes 2023-12-02T22:58:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2800 IOs, 1309631488 bytes 2023-12-02T22:58:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:2900 IOs, 1381649408 bytes 2023-12-02T22:58:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3000 IOs, 1450304512 bytes 2023-12-02T22:58:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3100 IOs, 1482516480 bytes 2023-12-02T22:58:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3200 IOs, 1505059840 bytes 2023-12-02T22:58:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3300 IOs, 1510666240 bytes 2023-12-02T22:58:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3400 IOs, 1573856256 bytes 2023-12-02T22:58:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3500 IOs, 1630112768 bytes 2023-12-02T22:58:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3600 IOs, 1706206208 bytes 2023-12-02T22:58:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3700 IOs, 1770297344 bytes 2023-12-02T22:58:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3800 IOs, 1780765696 bytes 2023-12-02T22:58:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:3900 IOs, 1827417088 bytes 2023-12-02T22:58:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4000 IOs, 1853466624 bytes 2023-12-02T22:58:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4100 IOs, 1923219456 bytes 2023-12-02T22:58:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4200 IOs, 1987309568 bytes 2023-12-02T22:58:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4300 IOs, 2060248064 bytes 2023-12-02T22:58:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4400 IOs, 2112318464 bytes 2023-12-02T22:58:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4500 IOs, 2169351168 bytes 2023-12-02T22:58:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4600 IOs, 2254158848 bytes 2023-12-02T22:58:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4700 IOs, 2286418944 bytes 2023-12-02T22:58:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4800 IOs, 2344636416 bytes 2023-12-02T22:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:4900 IOs, 2348228608 bytes 2023-12-02T22:58:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5000 IOs, 2369569792 bytes 2023-12-02T22:58:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5100 IOs, 2449374208 bytes 2023-12-02T22:58:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5200 IOs, 2491118592 bytes 2023-12-02T22:58:27.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-02T22:58:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5300 IOs, 2555348992 bytes 2023-12-02T22:58:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5400 IOs, 2567490560 bytes 2023-12-02T22:58:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5500 IOs, 2575561728 bytes 2023-12-02T22:58:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5600 IOs, 2578276352 bytes 2023-12-02T22:58:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5700 IOs, 2651332608 bytes 2023-12-02T22:58:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5800 IOs, 2724967424 bytes 2023-12-02T22:58:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:5900 IOs, 2756887552 bytes 2023-12-02T22:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6000 IOs, 2798717952 bytes 2023-12-02T22:58:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6100 IOs, 2819449856 bytes 2023-12-02T22:58:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6200 IOs, 2823576576 bytes 2023-12-02T22:58:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6300 IOs, 2853198848 bytes 2023-12-02T22:58:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6400 IOs, 2924440576 bytes 2023-12-02T22:58:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6500 IOs, 2976632832 bytes 2023-12-02T22:58:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6600 IOs, 3082161152 bytes 2023-12-02T22:58:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6700 IOs, 3137086464 bytes 2023-12-02T22:58:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6800 IOs, 3184765952 bytes 2023-12-02T22:58:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:6900 IOs, 3223236608 bytes 2023-12-02T22:58:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7000 IOs, 3245902848 bytes 2023-12-02T22:58:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7100 IOs, 3257717760 bytes 2023-12-02T22:58:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7200 IOs, 3289200640 bytes 2023-12-02T22:58:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7300 IOs, 3321573376 bytes 2023-12-02T22:58:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7400 IOs, 3388848128 bytes 2023-12-02T22:58:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7500 IOs, 3412005888 bytes 2023-12-02T22:58:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7600 IOs, 3437533184 bytes 2023-12-02T22:58:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7700 IOs, 3438155776 bytes 2023-12-02T22:58:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7800 IOs, 3483715584 bytes 2023-12-02T22:58:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:7900 IOs, 3498054656 bytes 2023-12-02T22:58:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8000 IOs, 3543443456 bytes 2023-12-02T22:58:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8100 IOs, 3577399296 bytes 2023-12-02T22:58:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8200 IOs, 3585995776 bytes 2023-12-02T22:58:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8300 IOs, 3609228288 bytes 2023-12-02T22:58:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8400 IOs, 3647858688 bytes 2023-12-02T22:58:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8500 IOs, 3648482304 bytes 2023-12-02T22:58:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8600 IOs, 3674098688 bytes 2023-12-02T22:58:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8700 IOs, 3732674560 bytes 2023-12-02T22:58:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8800 IOs, 3775165440 bytes 2023-12-02T22:58:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:8900 IOs, 3811551232 bytes 2023-12-02T22:58:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9000 IOs, 3901673472 bytes 2023-12-02T22:58:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9100 IOs, 3936529408 bytes 2023-12-02T22:58:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9200 IOs, 3947684864 bytes 2023-12-02T22:58:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9300 IOs, 3970136064 bytes 2023-12-02T22:58:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9400 IOs, 4018524160 bytes 2023-12-02T22:58:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9500 IOs, 4058003456 bytes 2023-12-02T22:58:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9600 IOs, 4168147968 bytes 2023-12-02T22:58:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9700 IOs, 4223945728 bytes 2023-12-02T22:58:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9800 IOs, 4297980928 bytes 2023-12-02T22:58:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:9900 IOs, 4362109952 bytes 2023-12-02T22:58:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10000 IOs, 4444735488 bytes 2023-12-02T22:58:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10100 IOs, 4486243328 bytes 2023-12-02T22:58:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10200 IOs, 4518163456 bytes 2023-12-02T22:58:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10300 IOs, 4541660160 bytes 2023-12-02T22:58:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10400 IOs, 4575439872 bytes 2023-12-02T22:58:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10500 IOs, 4653428736 bytes 2023-12-02T22:58:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10600 IOs, 4697743360 bytes 2023-12-02T22:58:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10700 IOs, 4715489280 bytes 2023-12-02T22:58:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10800 IOs, 4774510592 bytes 2023-12-02T22:58:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:10900 IOs, 4775838720 bytes 2023-12-02T22:58:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11000 IOs, 4802122752 bytes 2023-12-02T22:58:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11100 IOs, 4824774656 bytes 2023-12-02T22:58:32.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-02T22:58:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11200 IOs, 4869241856 bytes 2023-12-02T22:58:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11300 IOs, 4912272384 bytes 2023-12-02T22:58:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11400 IOs, 4949180416 bytes 2023-12-02T22:58:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11500 IOs, 4997386240 bytes 2023-12-02T22:58:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11600 IOs, 5037797376 bytes 2023-12-02T22:58:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11700 IOs, 5055286272 bytes 2023-12-02T22:58:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11800 IOs, 5126471680 bytes 2023-12-02T22:58:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:11900 IOs, 5141085184 bytes 2023-12-02T22:58:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12000 IOs, 5157421056 bytes 2023-12-02T22:58:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12100 IOs, 5181535232 bytes 2023-12-02T22:58:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12200 IOs, 5279557632 bytes 2023-12-02T22:58:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12300 IOs, 5335816192 bytes 2023-12-02T22:58:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12400 IOs, 5368576000 bytes 2023-12-02T22:58:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12500 IOs, 5387532288 bytes 2023-12-02T22:58:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12600 IOs, 5421155328 bytes 2023-12-02T22:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:12700 IOs, 5431971840 bytes 2023-12-02T22:58:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2023-12-02T22:58:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:58:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:58:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2023-12-02T22:58:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-02T22:58:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:58:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=15f84fe59f01 2023-12-02T22:58:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:58:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:58:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_10 present 15f84fe59f01 2023-12-02T22:58:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:58:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:58:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=15f84fe59f01 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2023-12-02T22:58:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-02T22:58:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=15f84fe59f01 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 15f84fe59f01 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 15f84fe59f01 -o 15f84fe59f01 = 15f84fe59f01 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_10 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:58:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-12-02T22:58:38.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-02T22:58:40.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:58:40.450+0000 7f20d9b20640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ff15852ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T22:58:40.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:58:40.450+0000 7f20e032d640 -1 rbd::mirror::ImageReplayer: 0x55ff1531e000 [3/1d37770c-44a0-4edf-937b-cf42e3d672b1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T22:58:43.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-02T22:58:49.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-02T22:58:54.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-02T22:58:59.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-02T22:59:01.358 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.35921830090314) > weight (0.25) 2023-12-02T22:59:01.358 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T22:59:01.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T22:59:01.359 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T22:59:01.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:01.358+0000 7fdb54cc6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 121673) UID: 1000 2023-12-02T22:59:05.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-02T22:59:05.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:05.964+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fca2f400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2023-12-02T22:59:05.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:05.964+0000 7fc38fc8f640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5651fca2f400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=46f07453-ee35-4496-a239-c8703d7c4316, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2023-12-02T22:59:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:59:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2023-12-02T22:59:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2023-12-02T22:59:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-02T22:59:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:59:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-02T22:59:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2023-12-02T22:59:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T22:59:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2023-12-02T22:59:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-12-02T22:59:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-02T22:59:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T22:59:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_10 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:59:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-12-02T22:59:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-02T22:59:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:59:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T22:59:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_10 2023-12-02T22:59:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2023-12-02T22:59:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-02T22:59:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-02T22:59:06.359 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T22:59:06.360 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T22:59:06.360 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T22:59:06.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T22:59:06.360 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T22:59:06.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T22:59:06.363 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-02T22:59:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T22:59:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2023-12-02T22:59:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.cQPVpMBksv/rbd-mirror.test.log 2023-12-02T22:59:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2023-12-02T22:59:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2023-12-02 22:59:06 TEST: check if removed 'images'\''' OMAP are removed 2023-12-02T22:59:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2023-12-02 22:59:06 TEST: check if removed images' OMAP are removed 2023-12-02T22:59:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-12-02T22:59:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-12-02T22:59:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2023-12-02T22:59:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=status_global 2023-12-02T22:59:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=status_global 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-02T22:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c status_global 2023-12-02T22:59:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-02T22:59:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:59:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_ 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_ 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-02T22:59:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_ 2023-12-02T22:59:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:59:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:59:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirror_leader 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_map 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-02T22:59:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2023-12-02T22:59:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-12-02T22:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 = 0 2023-12-02T22:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=1 2023-12-02T22:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 -eq 0 ']' 2023-12-02T22:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2023-12-02T22:59:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-02T22:59:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-12-02T22:59:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 = 0 2023-12-02T22:59:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=1 2023-12-02T22:59:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 -eq 0 ']' 2023-12-02T22:59:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2023-12-02T22:59:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-12-02T22:59:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2023-12-02T22:59:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:59:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=status_global 2023-12-02T22:59:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=status_global 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c status_global 2023-12-02T22:59:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-02T22:59:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:59:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_ 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_ 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-02T22:59:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_ 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirror_leader 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_map 2023-12-02T22:59:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2023-12-02T22:59:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cleanup 0 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code=0 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -ne 0 ']' 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z '' ']' 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster instance 2023-12-02T22:59:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:59:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-02T22:59:10.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-02T22:59:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' does not exist 2023-12-02T22:59:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:59:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-02T22:59:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' does not exist 2023-12-02T22:59:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:59:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-02T22:59:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' does not exist 2023-12-02T22:59:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2023-12-02T22:59:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-02T22:59:13.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:13.362+0000 7f8cb8a2f640 -1 rbd::mirror::LeaderWatcher: 0x561ee6b8a300 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-02T22:59:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:13.362+0000 7f8ca4206640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561ee66b0b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:13.362+0000 7f8ca4206640 -1 rbd::mirror::PoolWatcher: 0x561ee560ba40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:13.369+0000 7f8cb8a2f640 -1 rbd::mirror::LeaderWatcher: 0x561ee6b8a300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-02T22:59:13.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:13.369+0000 7f8cb8a2f640 -1 rbd::mirror::LeaderWatcher: 0x561ee6b8a300 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-02T22:59:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.402+0000 7fec6f335640 -1 rbd::mirror::LeaderWatcher: 0x5583f7f53b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-02T22:59:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec69329640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5583f8363200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec69329640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5583f83626c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec69329640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5583f80e2b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec68b28640 -1 rbd::mirror::PoolWatcher: 0x5583f7003a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec69329640 -1 rbd::mirror::PoolWatcher: 0x5583f70034a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec69329640 -1 rbd::mirror::PoolWatcher: 0x5583f7003860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec62b1c640 -1 rbd::mirror::PoolWatcher: 0x5583f7003c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec62b1c640 -1 rbd::mirror::PoolWatcher: 0x5583f7002b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.403+0000 7fec62b1c640 -1 rbd::mirror::PoolWatcher: 0x5583f70023c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:13.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.404+0000 7fec6f335640 -1 rbd::mirror::LeaderWatcher: 0x5583f7f53b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-02T22:59:13.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:13.404+0000 7fec6f335640 -1 rbd::mirror::LeaderWatcher: 0x5583f7f53b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-02T22:59:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.555+0000 7fd88b561640 -1 rbd::mirror::RemotePoolPoller: 0x562b7bb25790 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-02T22:59:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.556+0000 7fd88bd62640 -1 rbd::mirror::RemotePoolPoller: 0x562b7bb25790 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-02T22:59:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.556+0000 7fd88bd62640 -1 rbd::mirror::PoolReplayer: 0x562b7c9ac000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-02T22:59:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.556+0000 7fd89857b640 -1 rbd::mirror::InstanceWatcher: 0x562b7ccaee00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6591 object: (2) No such file or directory 2023-12-02T22:59:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.556+0000 7fd89857b640 -1 rbd::mirror::InstanceWatcher: 0x562b7ccc4700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6591 object: (2) No such file or directory 2023-12-02T22:59:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.556+0000 7fd891d6e640 -1 rbd::mirror::InstanceWatcher: 0x562b7ccaee00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.556+0000 7fd891d6e640 -1 rbd::mirror::InstanceWatcher: 0x562b7ccc4700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd88b561640 -1 rbd::mirror::MirrorStatusUpdater 0x562b7cd067e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd88b561640 -1 rbd::mirror::MirrorStatusUpdater 0x562b7cd06360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd89857b640 -1 rbd::mirror::NamespaceReplayer: 0x562b7cca6820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd89857b640 -1 rbd::mirror::NamespaceReplayer: 0x562b7cca69c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd89256f640 -1 rbd::mirror::MirrorStatusUpdater 0x562b7cd06000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd89256f640 -1 rbd::mirror::MirrorStatusUpdater 0x562b7cd06120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd898d7c640 -1 rbd::mirror::NamespaceReplayer: 0x562b7cca6820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.557+0000 7fd898d7c640 -1 rbd::mirror::NamespaceReplayer: 0x562b7cca69c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:13.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.672+0000 7fd87dd46640 -1 rbd::mirror::RemotePoolPoller: 0x562b7bb25ee0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-02T22:59:13.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.673+0000 7fd87dd46640 -1 rbd::mirror::RemotePoolPoller: 0x562b7bb25ee0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-02T22:59:13.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.673+0000 7fd87dd46640 -1 rbd::mirror::PoolReplayer: 0x562b7ccca000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-02T22:59:13.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.732+0000 7fd89857b640 -1 rbd::mirror::LeaderWatcher: 0x562b7ca9db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:13.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.732+0000 7fd89857b640 -1 rbd::mirror::InstanceWatcher: 0x562b7c991880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6591 object: (2) No such file or directory 2023-12-02T22:59:13.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.733+0000 7fd89256f640 -1 rbd::mirror::InstanceWatcher: 0x562b7c991880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.733+0000 7fd88bd62640 -1 rbd::mirror::MirrorStatusUpdater 0x562b7ca806c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.733+0000 7fd89857b640 -1 rbd::mirror::NamespaceReplayer: 0x562b7cca6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.733+0000 7fd891d6e640 -1 rbd::mirror::MirrorStatusUpdater 0x562b7d5eeb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.733+0000 7fd898d7c640 -1 rbd::mirror::NamespaceReplayer: 0x562b7cca6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:13.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.735+0000 7fd89cc1b640 -1 rbd::mirror::Mirror: 0x562b7bb7fd40 update_pool_replayers: restarting failed pool replayer for uuid: 91241553-4190-45ba-be25-7142ffbdf931 cluster: cluster1 client: client.mirror 2023-12-02T22:59:13.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:13.899+0000 7fd89cc1b640 -1 rbd::mirror::PoolReplayer: 0x562b7ccca000 init: error accessing remote pool mirror_parent: (2) No such file or directory 2023-12-02T22:59:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' does not exist 2023-12-02T22:59:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-02T22:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirrors cluster1 2023-12-02T22:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2023-12-02T22:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 3 2023-12-02T22:59:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster1:0 '' 2023-12-02T22:59:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2023-12-02T22:59:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster1:1 '' 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster1:2 '' 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:2 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster1:3 '' 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:3 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirrors cluster2 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 3 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster2:0 '' 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:0 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster2:1 '' 2023-12-02T22:59:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:1 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster2:2 '' 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:2 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster2:3 '' 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:3 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 1 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z 1 ']' 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test '' = /tmp/tmp.cQPVpMBksv 2023-12-02T22:59:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -Rf /tmp/tmp.cQPVpMBksv 2023-12-02T22:59:14.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.431+0000 7fc39cca9640 -1 rbd::mirror::InstanceWatcher: 0x5651fbe0e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5624 object: (2) No such file or directory 2023-12-02T22:59:14.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.431+0000 7fc39cca9640 -1 rbd::mirror::InstanceWatcher: 0x5651fbebc000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5624 object: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.431+0000 7fc39649c640 -1 rbd::mirror::InstanceWatcher: 0x5651fbe0e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.431+0000 7fc39649c640 -1 rbd::mirror::InstanceWatcher: 0x5651fbebc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc38fc8f640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fbe625a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc38fc8f640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fbe62360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc39cca9640 -1 rbd::mirror::NamespaceReplayer: 0x5651fbcc2820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc39cca9640 -1 rbd::mirror::NamespaceReplayer: 0x5651fbcc29c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc395c9b640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fbe62000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc395c9b640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fbe62120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc39c4a8640 -1 rbd::mirror::NamespaceReplayer: 0x5651fbcc2820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.432+0000 7fc39c4a8640 -1 rbd::mirror::NamespaceReplayer: 0x5651fbcc29c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:OK 2023-12-02T22:59:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2023-12-02T22:59:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo OK 2023-12-02T22:59:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ exit 0 2023-12-02T22:59:14.453 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-02T22:59:14.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-02T22:59:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.468+0000 7fc38f48e640 -1 rbd::mirror::RemotePoolPoller: 0x5651facb55f0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-02T22:59:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.468+0000 7fc38f48e640 -1 rbd::mirror::RemotePoolPoller: 0x5651facb55f0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-02T22:59:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.468+0000 7fc38f48e640 -1 rbd::mirror::PoolReplayer: 0x5651fbb24000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-02T22:59:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.517+0000 7fc381c73640 -1 rbd::mirror::RemotePoolPoller: 0x5651facb5e10 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-02T22:59:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.517+0000 7fc381c73640 -1 rbd::mirror::RemotePoolPoller: 0x5651facb5e10 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-02T22:59:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.517+0000 7fc381c73640 -1 rbd::mirror::PoolReplayer: 0x5651fbe28000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-02T22:59:14.582 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-02T22:59:14.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-02T22:59:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.611+0000 7fc39cca9640 -1 rbd::mirror::LeaderWatcher: 0x5651fbbddb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.611+0000 7fc39cca9640 -1 rbd::mirror::InstanceWatcher: 0x5651fbb0f880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5624 object: (2) No such file or directory 2023-12-02T22:59:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.611+0000 7fc395c9b640 -1 rbd::mirror::InstanceWatcher: 0x5651fbb0f880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.612+0000 7fc38f48e640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fbbc86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.612+0000 7fc39cca9640 -1 rbd::mirror::NamespaceReplayer: 0x5651fbcc2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.612+0000 7fc395c9b640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fc778a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.612+0000 7fc39c4a8640 -1 rbd::mirror::NamespaceReplayer: 0x5651fbcc2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.614+0000 7fc39cca9640 -1 rbd::mirror::LeaderWatcher: 0x5651fc224300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.615+0000 7fc39cca9640 -1 rbd::mirror::InstanceWatcher: 0x5651fbe0e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5627 object: (2) No such file or directory 2023-12-02T22:59:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.615+0000 7fc388c81640 -1 rbd::mirror::InstanceWatcher: 0x5651fbe0e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.615+0000 7fc388480640 -1 rbd::mirror::MirrorStatusUpdater 0x5651fc046a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:14.615+0000 7fc39cca9640 -1 rbd::mirror::NamespaceReplayer: 0x5651fc182680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:14.816 DEBUG:teuthology.parallel:result is None 2023-12-02T22:59:14.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-02T22:59:14.883 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-02T22:59:14.884 DEBUG:teuthology.orchestra.run.smithi170:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T22:59:14.942 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-02T22:59:14.942 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-02T22:59:14.953 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-02T22:59:14.953 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2023-12-02T22:59:14.953 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-02T22:59:14.954 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-02T22:59:14.963 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-02T22:59:14.963 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2023-12-02T22:59:14.963 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-02T22:59:14.963 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.006 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-02T22:59:15.006 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.005+0000 7f20e0b2e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 109628) UID: 1000 2023-12-02T22:59:15.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20dfb2c640 -1 rbd::mirror::LeaderWatcher: 0x55ff150ee300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20d931f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ff15235680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20d931f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ff14f42000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20d931f640 -1 rbd::mirror::PoolWatcher: 0x55ff13bfeb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20d931f640 -1 rbd::mirror::PoolWatcher: 0x55ff13bfe3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20e032d640 -1 rbd::mirror::InstanceWatcher: 0x55ff151fc000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5590 object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20d2b12640 -1 rbd::mirror::PoolWatcher: 0x55ff1543c000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20d2b12640 -1 rbd::mirror::PoolWatcher: 0x55ff13bffe00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20cc305640 -1 rbd::mirror::InstanceWatcher: 0x55ff151fc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20e032d640 -1 rbd::mirror::InstanceWatcher: 0x55ff14d8d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5587 object: (2) No such file or directory 2023-12-02T22:59:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.006+0000 7f20dfb2c640 -1 rbd::mirror::InstanceWatcher: 0x55ff15230700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5587 object: (2) No such file or directory 2023-12-02T22:59:15.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20d9b20640 -1 rbd::mirror::InstanceWatcher: 0x55ff14d8d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20d931f640 -1 rbd::mirror::InstanceWatcher: 0x55ff15230700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20c5af8640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14dc26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20dfb2c640 -1 rbd::mirror::NamespaceReplayer: 0x55ff150dc820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20cc305640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14fae000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20e032d640 -1 rbd::mirror::NamespaceReplayer: 0x55ff150dc820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20d2b12640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14dc25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20dfb2c640 -1 rbd::mirror::NamespaceReplayer: 0x55ff14d68820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20d3313640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14dc2360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20dfb2c640 -1 rbd::mirror::NamespaceReplayer: 0x55ff14d689c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20d9b20640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14dc2000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20d9b20640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14dc2120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20e032d640 -1 rbd::mirror::NamespaceReplayer: 0x55ff14d68820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.007+0000 7f20e032d640 -1 rbd::mirror::NamespaceReplayer: 0x55ff14d689c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.009+0000 7f20d9b20640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ff15234480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.009+0000 7f20d931f640 -1 rbd::mirror::PoolWatcher: 0x55ff13bffc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.009+0000 7f20d2b12640 -1 rbd::mirror::PoolWatcher: 0x55ff1543c1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20dfb2c640 -1 rbd::mirror::LeaderWatcher: 0x55ff14b3bb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20dfb2c640 -1 rbd::mirror::LeaderWatcher: 0x55ff14b3bb00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20e032d640 -1 rbd::mirror::LeaderWatcher: 0x55ff14b3bb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20e032d640 -1 rbd::mirror::InstanceWatcher: 0x55ff14a69880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5587 object: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20d931f640 -1 rbd::mirror::InstanceWatcher: 0x55ff14a69880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20d2b12640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14b4a5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.010+0000 7f20e032d640 -1 rbd::mirror::NamespaceReplayer: 0x55ff14d68000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.011+0000 7f20d9b20640 -1 rbd::mirror::MirrorStatusUpdater 0x55ff14ae2b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.011+0000 7f20dfb2c640 -1 rbd::mirror::NamespaceReplayer: 0x55ff14d68000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-12-02T22:59:15.024 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.049 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-02T22:59:15.049 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.064+0000 7faf274e3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 112069) UID: 1000 2023-12-02T22:59:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.064+0000 7faf124b9640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ee40216d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-02T22:59:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf26ce2640 -1 rbd::mirror::InstanceWatcher: 0x55ee3fc67500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5717 object: (2) No such file or directory 2023-12-02T22:59:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf26ce2640 -1 rbd::mirror::InstanceWatcher: 0x55ee3fcee380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5717 object: (2) No such file or directory 2023-12-02T22:59:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf124b9640 -1 rbd::mirror::PoolWatcher: 0x55ee3eadfe00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf204d5640 -1 rbd::mirror::InstanceWatcher: 0x55ee3fc67500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf1fcd4640 -1 rbd::mirror::InstanceWatcher: 0x55ee3fcee380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf0bcac640 -1 rbd::mirror::PoolWatcher: 0x55ee40414000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf19cc8640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fcba360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf19cc8640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fcba5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf264e1640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3fc64820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf264e1640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3fc649c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf204d5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fcba000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf204d5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fcba120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf26ce2640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3fc64820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf26ce2640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3fc649c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf26ce2640 -1 rbd::mirror::LeaderWatcher: 0x55ee40082300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.065+0000 7faf26ce2640 -1 rbd::mirror::LeaderWatcher: 0x55ee40082300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.066+0000 7faf264e1640 -1 rbd::mirror::LeaderWatcher: 0x55ee40082300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.066+0000 7faf264e1640 -1 rbd::mirror::InstanceWatcher: 0x55ee3fda3180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5720 object: (2) No such file or directory 2023-12-02T22:59:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.066+0000 7faf12cba640 -1 rbd::mirror::InstanceWatcher: 0x55ee3fda3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.067+0000 7faf0c4ad640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fec07e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.067+0000 7faf264e1640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3ffe4680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.067+0000 7faf124b9640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3feb46c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.067+0000 7faf26ce2640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3ffe4680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.069+0000 7faf264e1640 -1 rbd::mirror::LeaderWatcher: 0x55ee3fa21b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.069+0000 7faf264e1640 -1 rbd::mirror::InstanceWatcher: 0x55ee3f963880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5717 object: (2) No such file or directory 2023-12-02T22:59:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.069+0000 7faf1fcd4640 -1 rbd::mirror::InstanceWatcher: 0x55ee3f963880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.070+0000 7faf194c7640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fa405a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.070+0000 7faf264e1640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3fc64000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.070+0000 7faf204d5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee3fa1ac60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.070+0000 7faf26ce2640 -1 rbd::mirror::NamespaceReplayer: 0x55ee3fc64000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-12-02T22:59:15.083 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.093 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-12-02T22:59:15.093 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.123+0000 7f162931a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 126842) UID: 1000 2023-12-02T22:59:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.123+0000 7f1628b19640 -1 rbd::mirror::LeaderWatcher: 0x56311e6ce300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.123+0000 7f1628b19640 -1 rbd::mirror::InstanceWatcher: 0x56311e2cf880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6642 object: (2) No such file or directory 2023-12-02T22:59:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.123+0000 7f1628b19640 -1 rbd::mirror::InstanceWatcher: 0x56311e2cee00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6642 object: (2) No such file or directory 2023-12-02T22:59:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1628b19640 -1 rbd::mirror::InstanceWatcher: 0x56311e2cea80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6645 object: (2) No such file or directory 2023-12-02T22:59:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f162230c640 -1 rbd::mirror::InstanceWatcher: 0x56311e2cf880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1621b0b640 -1 rbd::mirror::InstanceWatcher: 0x56311e2cee00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f16142f0640 -1 rbd::mirror::InstanceWatcher: 0x56311e2cea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f161b2fe640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e30e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f161b2fe640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e30e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1628318640 -1 rbd::mirror::NamespaceReplayer: 0x56311e2c2820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1628318640 -1 rbd::mirror::NamespaceReplayer: 0x56311e2c29c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f160e2e4640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e5526c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1628b19640 -1 rbd::mirror::NamespaceReplayer: 0x56311e62a680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1621b0b640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e30e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.124+0000 7f1621b0b640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e30e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.125+0000 7f1628318640 -1 rbd::mirror::NamespaceReplayer: 0x56311e2c2820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.125+0000 7f1628318640 -1 rbd::mirror::NamespaceReplayer: 0x56311e2c29c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.125+0000 7f1614af1640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e50c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.125+0000 7f1628b19640 -1 rbd::mirror::NamespaceReplayer: 0x56311e62a680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.126+0000 7f1628318640 -1 rbd::mirror::LeaderWatcher: 0x56311e095b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.127+0000 7f1628318640 -1 rbd::mirror::InstanceWatcher: 0x56311dfb3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6642 object: (2) No such file or directory 2023-12-02T22:59:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.127+0000 7f162230c640 -1 rbd::mirror::InstanceWatcher: 0x56311dfb3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.127+0000 7f161b2fe640 -1 rbd::mirror::MirrorStatusUpdater 0x56311e07e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.127+0000 7f1628318640 -1 rbd::mirror::NamespaceReplayer: 0x56311e2c2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.127+0000 7f1621b0b640 -1 rbd::mirror::MirrorStatusUpdater 0x56311ec14b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.127+0000 7f1628b19640 -1 rbd::mirror::NamespaceReplayer: 0x56311e2c2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-12-02T22:59:15.140 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.151 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-12-02T22:59:15.151 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi170.stderr:2023-12-02T22:59:15.180+0000 7fd89957d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 126208) UID: 1000 2023-12-02T22:59:15.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-12-02T22:59:15.191 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.202 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-02T22:59:15.202 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.231+0000 7f87bfbe8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 126379) UID: 1000 2023-12-02T22:59:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.231+0000 7f87bebe6640 -1 rbd::mirror::LeaderWatcher: 0x56346978e300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87bf3e7640 -1 rbd::mirror::InstanceWatcher: 0x563469370e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6151 object: (2) No such file or directory 2023-12-02T22:59:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87bf3e7640 -1 rbd::mirror::InstanceWatcher: 0x5634693ce700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6151 object: (2) No such file or directory 2023-12-02T22:59:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87bebe6640 -1 rbd::mirror::InstanceWatcher: 0x56346983e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6153 object: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87b83d9640 -1 rbd::mirror::InstanceWatcher: 0x563469370e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87b83d9640 -1 rbd::mirror::InstanceWatcher: 0x5634693ce700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87ab3bf640 -1 rbd::mirror::InstanceWatcher: 0x56346983e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87b23cd640 -1 rbd::mirror::MirrorStatusUpdater 0x5634693c4360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87b23cd640 -1 rbd::mirror::MirrorStatusUpdater 0x5634693c45a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87bf3e7640 -1 rbd::mirror::NamespaceReplayer: 0x563469352820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.232+0000 7f87bf3e7640 -1 rbd::mirror::NamespaceReplayer: 0x5634693529c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87a4bb2640 -1 rbd::mirror::MirrorStatusUpdater 0x5634695ca7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87bebe6640 -1 rbd::mirror::NamespaceReplayer: 0x5634697924e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87b83d9640 -1 rbd::mirror::MirrorStatusUpdater 0x5634693c4000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87b83d9640 -1 rbd::mirror::MirrorStatusUpdater 0x5634693c4120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87bf3e7640 -1 rbd::mirror::NamespaceReplayer: 0x563469352820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87bf3e7640 -1 rbd::mirror::NamespaceReplayer: 0x5634693529c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87aabbe640 -1 rbd::mirror::MirrorStatusUpdater 0x5634695a27e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.233+0000 7f87bebe6640 -1 rbd::mirror::NamespaceReplayer: 0x5634697924e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.235+0000 7f87bf3e7640 -1 rbd::mirror::LeaderWatcher: 0x563469145b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.235+0000 7f87bf3e7640 -1 rbd::mirror::InstanceWatcher: 0x563468f7d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6151 object: (2) No such file or directory 2023-12-02T22:59:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.235+0000 7f87b83d9640 -1 rbd::mirror::InstanceWatcher: 0x563468f7d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.235+0000 7f87b1bcc640 -1 rbd::mirror::MirrorStatusUpdater 0x5634690e06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.235+0000 7f87bf3e7640 -1 rbd::mirror::NamespaceReplayer: 0x563469352000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.236+0000 7f87b8bda640 -1 rbd::mirror::MirrorStatusUpdater 0x5634690deb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi170.stderr:2023-12-02T22:59:15.236+0000 7f87bebe6640 -1 rbd::mirror::NamespaceReplayer: 0x563469352000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-12-02T22:59:15.248 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.259 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-12-02T22:59:15.259 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.288+0000 7f8cb9230640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 123440) UID: 1000 2023-12-02T22:59:15.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cb8a2f640 -1 rbd::mirror::LeaderWatcher: 0x561ee6b8a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cb8a2f640 -1 rbd::mirror::InstanceWatcher: 0x561ee6774380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5995 object: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cb8a2f640 -1 rbd::mirror::InstanceWatcher: 0x561ee67ee700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5995 object: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cb822e640 -1 rbd::mirror::InstanceWatcher: 0x561ee67ef880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5998 object: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cb1a21640 -1 rbd::mirror::InstanceWatcher: 0x561ee6774380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cb1a21640 -1 rbd::mirror::InstanceWatcher: 0x561ee67ee700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8ca4a07640 -1 rbd::mirror::InstanceWatcher: 0x561ee67ef880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.289+0000 7f8cab214640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee67bc480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cab214640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee67bc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb8a2f640 -1 rbd::mirror::NamespaceReplayer: 0x561ee6772820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb8a2f640 -1 rbd::mirror::NamespaceReplayer: 0x561ee67729c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8c9e1fa640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee67bc900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb822e640 -1 rbd::mirror::NamespaceReplayer: 0x561ee6ae8680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb2222640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee67bc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb822e640 -1 rbd::mirror::NamespaceReplayer: 0x561ee6772820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb2222640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee67bc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb8a2f640 -1 rbd::mirror::NamespaceReplayer: 0x561ee67729c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8ca4206640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee699ca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.290+0000 7f8cb822e640 -1 rbd::mirror::NamespaceReplayer: 0x561ee6ae8680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.292+0000 7f8cb8a2f640 -1 rbd::mirror::LeaderWatcher: 0x561ee6535b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.292+0000 7f8cb8a2f640 -1 rbd::mirror::InstanceWatcher: 0x561ee6475880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5995 object: (2) No such file or directory 2023-12-02T22:59:15.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.293+0000 7f8cb2222640 -1 rbd::mirror::InstanceWatcher: 0x561ee6475880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.294+0000 7f8caba15640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee65206c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.295+0000 7f8cb8a2f640 -1 rbd::mirror::NamespaceReplayer: 0x561ee6772000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.295+0000 7f8cb2222640 -1 rbd::mirror::MirrorStatusUpdater 0x561ee70e0a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi170.stderr:2023-12-02T22:59:15.295+0000 7f8cb822e640 -1 rbd::mirror::NamespaceReplayer: 0x561ee6772000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-12-02T22:59:15.308 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.320 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-02T22:59:15.320 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.348+0000 7fec70337640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 124171) UID: 1000 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.348+0000 7fec6f335640 -1 rbd::mirror::LeaderWatcher: 0x5583f8590300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6f335640 -1 rbd::mirror::InstanceWatcher: 0x5583f8184700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6029 object: (2) No such file or directory 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6f335640 -1 rbd::mirror::InstanceWatcher: 0x5583f819b500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6029 object: (2) No such file or directory 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6f335640 -1 rbd::mirror::InstanceWatcher: 0x5583f8648380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6032 object: (2) No such file or directory 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec69329640 -1 rbd::mirror::InstanceWatcher: 0x5583f8184700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec69329640 -1 rbd::mirror::InstanceWatcher: 0x5583f819b500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec5bb0e640 -1 rbd::mirror::InstanceWatcher: 0x5583f8648380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6231b640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f81c0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6231b640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f81c05a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6fb36640 -1 rbd::mirror::NamespaceReplayer: 0x5583f8182820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6fb36640 -1 rbd::mirror::NamespaceReplayer: 0x5583f81829c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec69329640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f81c0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec55301640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f83d67e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec69329640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f81c0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6f335640 -1 rbd::mirror::NamespaceReplayer: 0x5583f8182820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6f335640 -1 rbd::mirror::NamespaceReplayer: 0x5583f84f4680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.349+0000 7fec6f335640 -1 rbd::mirror::NamespaceReplayer: 0x5583f81829c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.350+0000 7fec5b30d640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f83a87e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.350+0000 7fec6fb36640 -1 rbd::mirror::NamespaceReplayer: 0x5583f84f4680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.352+0000 7fec6f335640 -1 rbd::mirror::LeaderWatcher: 0x5583f7f53b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-02T22:59:15.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.352+0000 7fec6f335640 -1 rbd::mirror::InstanceWatcher: 0x5583f7da5880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6029 object: (2) No such file or directory 2023-12-02T22:59:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.352+0000 7fec68b28640 -1 rbd::mirror::InstanceWatcher: 0x5583f7da5880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-02T22:59:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.352+0000 7fec62b1c640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f7f3e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.352+0000 7fec6f335640 -1 rbd::mirror::NamespaceReplayer: 0x5583f8182000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.353+0000 7fec69329640 -1 rbd::mirror::MirrorStatusUpdater 0x5583f7edeb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-02T22:59:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi170.stderr:2023-12-02T22:59:15.353+0000 7fec6fb36640 -1 rbd::mirror::NamespaceReplayer: 0x5583f8182000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-02T22:59:15.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-12-02T22:59:15.366 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T22:59:15.377 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-12-02T22:59:15.378 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T22:59:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi170.stderr:2023-12-02T22:59:15.406+0000 7fc39d4aa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 116084) UID: 1000 2023-12-02T22:59:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-12-02T22:59:15.418 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-02T22:59:15.429 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-02T22:59:15.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:15.755 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:15.756 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:15.768 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2023-12-02T22:59:14.892869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4828968,"kb_used_data":378732,"kb_used_omap":276,"kb_used_meta":4449451,"kb_avail":278286552,"statfs":{"total":289910292480,"available":284965429248,"internally_reserved":0,"allocated":387821568,"data_stored":378774818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4556237891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536861868,"num_objects":-162,"num_object_clones":0,"num_object_copies":-324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-162,"num_whiteouts":0,"num_read":-143246,"num_read_kb":-39598293,"num_write":-556987,"num_write_kb":-200975720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9188352,"num_keys_recovered":-9,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.949520+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:04.949520+0000","last_peered":"2023-12-02T22:59:04.949520+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:04.949520+0000","last_undegraded":"2023-12-02T22:59:04.949520+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.950379+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:04.950379+0000","last_peered":"2023-12-02T22:59:04.950379+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:04.950379+0000","last_undegraded":"2023-12-02T22:59:04.950379+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:05.483420+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:05.483420+0000","last_peered":"2023-12-02T22:59:05.483420+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:05.483420+0000","last_undegraded":"2023-12-02T22:59:05.483420+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:05.483261+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:05.483261+0000","last_peered":"2023-12-02T22:59:05.483261+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:05.483261+0000","last_undegraded":"2023-12-02T22:59:05.483261+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.949656+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:04.949656+0000","last_peered":"2023-12-02T22:59:04.949656+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:04.949656+0000","last_undegraded":"2023-12-02T22:59:04.949656+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":498,"state":"active+clean","last_fresh":"2023-12-02T22:59:05.483233+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:05.483233+0000","last_peered":"2023-12-02T22:59:05.483233+0000","last_clean":"2023-12-02T22:59:05.483233+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:05.483233+0000","last_undegraded":"2023-12-02T22:59:05.483233+0000","last_fullsized":"2023-12-02T22:59:05.483233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":647,"reported_epoch":498,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:04.948486+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:04.948486+0000","last_peered":"2023-12-02T22:59:04.948486+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:04.948486+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483264,"kb_used_data":5812,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888576,"statfs":{"total":96636764160,"available":95117901824,"internally_reserved":0,"allocated":5951488,"data_stored":2970922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1471796,"kb_used_data":6376,"kb_used_omap":92,"kb_used_meta":1465315,"kb_avail":92900044,"statfs":{"total":96636764160,"available":95129645056,"internally_reserved":0,"allocated":6529024,"data_stored":3554084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1500482802},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.246},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:15.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:16.045 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:16.057 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2023-12-02T22:59:14.892869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4828968,"kb_used_data":378732,"kb_used_omap":276,"kb_used_meta":4449451,"kb_avail":278286552,"statfs":{"total":289910292480,"available":284965429248,"internally_reserved":0,"allocated":387821568,"data_stored":378774818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4556237891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536861868,"num_objects":-162,"num_object_clones":0,"num_object_copies":-324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-162,"num_whiteouts":0,"num_read":-143246,"num_read_kb":-39598293,"num_write":-556987,"num_write_kb":-200975720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9188352,"num_keys_recovered":-9,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.949520+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:04.949520+0000","last_peered":"2023-12-02T22:59:04.949520+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:04.949520+0000","last_undegraded":"2023-12-02T22:59:04.949520+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.950379+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:04.950379+0000","last_peered":"2023-12-02T22:59:04.950379+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:04.950379+0000","last_undegraded":"2023-12-02T22:59:04.950379+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:05.483420+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:05.483420+0000","last_peered":"2023-12-02T22:59:05.483420+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:05.483420+0000","last_undegraded":"2023-12-02T22:59:05.483420+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:05.483261+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:05.483261+0000","last_peered":"2023-12-02T22:59:05.483261+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:05.483261+0000","last_undegraded":"2023-12-02T22:59:05.483261+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.949656+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:04.949656+0000","last_peered":"2023-12-02T22:59:04.949656+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:04.949656+0000","last_undegraded":"2023-12-02T22:59:04.949656+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":498,"state":"active+clean","last_fresh":"2023-12-02T22:59:05.483233+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:05.483233+0000","last_peered":"2023-12-02T22:59:05.483233+0000","last_clean":"2023-12-02T22:59:05.483233+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:05.483233+0000","last_undegraded":"2023-12-02T22:59:05.483233+0000","last_fullsized":"2023-12-02T22:59:05.483233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":647,"reported_epoch":498,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:04.948486+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:04.948486+0000","last_peered":"2023-12-02T22:59:04.948486+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:04.948486+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483264,"kb_used_data":5812,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888576,"statfs":{"total":96636764160,"available":95117901824,"internally_reserved":0,"allocated":5951488,"data_stored":2970922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1471796,"kb_used_data":6376,"kb_used_omap":92,"kb_used_meta":1465315,"kb_avail":92900044,"statfs":{"total":96636764160,"available":95129645056,"internally_reserved":0,"allocated":6529024,"data_stored":3554084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1500482802},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.246},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:16.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:16.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-02T22:59:16.333 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:16.334 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:16.346 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2023-12-02T22:59:14.892869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4828968,"kb_used_data":378732,"kb_used_omap":276,"kb_used_meta":4449451,"kb_avail":278286552,"statfs":{"total":289910292480,"available":284965429248,"internally_reserved":0,"allocated":387821568,"data_stored":378774818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4556237891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536861868,"num_objects":-162,"num_object_clones":0,"num_object_copies":-324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-162,"num_whiteouts":0,"num_read":-143246,"num_read_kb":-39598293,"num_write":-556987,"num_write_kb":-200975720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9188352,"num_keys_recovered":-9,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.949520+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:04.949520+0000","last_peered":"2023-12-02T22:59:04.949520+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:04.949520+0000","last_undegraded":"2023-12-02T22:59:04.949520+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.950379+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:04.950379+0000","last_peered":"2023-12-02T22:59:04.950379+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:04.950379+0000","last_undegraded":"2023-12-02T22:59:04.950379+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:05.483420+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:05.483420+0000","last_peered":"2023-12-02T22:59:05.483420+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:05.483420+0000","last_undegraded":"2023-12-02T22:59:05.483420+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:05.483261+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:05.483261+0000","last_peered":"2023-12-02T22:59:05.483261+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:05.483261+0000","last_undegraded":"2023-12-02T22:59:05.483261+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":498,"state":"active+undersized","last_fresh":"2023-12-02T22:59:04.949656+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:04.949656+0000","last_peered":"2023-12-02T22:59:04.949656+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:04.949656+0000","last_undegraded":"2023-12-02T22:59:04.949656+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":498,"state":"active+clean","last_fresh":"2023-12-02T22:59:05.483233+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:05.483233+0000","last_peered":"2023-12-02T22:59:05.483233+0000","last_clean":"2023-12-02T22:59:05.483233+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:05.483233+0000","last_undegraded":"2023-12-02T22:59:05.483233+0000","last_fullsized":"2023-12-02T22:59:05.483233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":647,"reported_epoch":498,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:04.948486+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:04.948486+0000","last_peered":"2023-12-02T22:59:04.948486+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:04.948486+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483264,"kb_used_data":5812,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888576,"statfs":{"total":96636764160,"available":95117901824,"internally_reserved":0,"allocated":5951488,"data_stored":2970922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1471796,"kb_used_data":6376,"kb_used_omap":92,"kb_used_meta":1465315,"kb_avail":92900044,"statfs":{"total":96636764160,"available":95129645056,"internally_reserved":0,"allocated":6529024,"data_stored":3554084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1500482802},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.246},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:16.347 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:16.347 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 498, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:04.949520+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:04.949520+0000', 'last_peered': '2023-12-02T22:59:04.949520+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:04.949520+0000', 'last_undegraded': '2023-12-02T22:59:04.949520+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:16.347 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:16.347 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 498, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:04.950379+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:04.950379+0000', 'last_peered': '2023-12-02T22:59:04.950379+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:04.950379+0000', 'last_undegraded': '2023-12-02T22:59:04.950379+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 498, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:05.483420+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:05.483420+0000', 'last_peered': '2023-12-02T22:59:05.483420+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:05.483420+0000', 'last_undegraded': '2023-12-02T22:59:05.483420+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 498, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:05.483261+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:05.483261+0000', 'last_peered': '2023-12-02T22:59:05.483261+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:05.483261+0000', 'last_undegraded': '2023-12-02T22:59:05.483261+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 498, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:04.949656+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:04.949656+0000', 'last_peered': '2023-12-02T22:59:04.949656+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:04.949656+0000', 'last_undegraded': '2023-12-02T22:59:04.949656+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:16.348 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 647, 'reported_epoch': 498, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:04.948486+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:04.948486+0000', 'last_peered': '2023-12-02T22:59:04.948486+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:04.948486+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:16.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:16.688 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:16.688 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2281,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4944863232,"bytes_avail":284965429248,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":45,"modified":"2023-12-02T22:59:06.889921+0000","services":{"rbd-mirror":{"daemons":{"summary":"","5584":{"start_epoch":31,"start_stamp":"2023-12-02T22:45:10.404869+0000","gid":5584,"addr":"172.21.15.170:0/441170683","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev)","ceph_version_short":"18.0.0-7670-g0c523c70","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi170","id":"mirror.3","instance_id":"5584","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023","kernel_version":"5.14.0-388.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488648","os":"Linux"},"task_status":{}},"5714":{"start_epoch":32,"start_stamp":"2023-12-02T22:46:26.422054+0000","gid":5714,"addr":"172.21.15.170:0/4223007082","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev)","ceph_version_short":"18.0.0-7670-g0c523c70","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi170","id":"mirror.2","instance_id":"5714","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023","kernel_version":"5.14.0-388.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488648","os":"Linux"},"task_status":{}},"6588":{"start_epoch":43,"start_stamp":"2023-12-02T22:57:43.475800+0000","gid":6588,"addr":"172.21.15.170:0/1737770509","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev)","ceph_version_short":"18.0.0-7670-g0c523c70","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi170","id":"mirror.0","instance_id":"6588","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023","kernel_version":"5.14.0-388.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488648","os":"Linux"},"task_status":{}},"6640":{"start_epoch":45,"start_stamp":"2023-12-02T22:59:06.476993+0000","gid":6640,"addr":"172.21.15.170:0/2016412137","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.0.0-7670-g0c523c70 (0c523c70ca7aa7b49aa5b461902b17d9e90c1f95) reef (dev)","ceph_version_short":"18.0.0-7670-g0c523c70","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi170","id":"mirror.1","instance_id":"6640","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023","kernel_version":"5.14.0-388.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488648","os":"Linux"},"task_status":{}}}}}},"progress_events":{}} 2023-12-02T22:59:16.701 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:16.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:16.979 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:16.979 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:16.991 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2023-12-02T22:59:16.893329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847044,"kb_used_data":378760,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268476,"statfs":{"total":289910292480,"available":284946919424,"internally_reserved":0,"allocated":387850240,"data_stored":378788737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536861868,"num_objects":-162,"num_object_clones":0,"num_object_copies":-324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-162,"num_whiteouts":0,"num_read":-143246,"num_read_kb":-39598293,"num_write":-556987,"num_write_kb":-200975720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9188352,"num_keys_recovered":-9,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002696"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483264,"kb_used_data":5812,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888576,"statfs":{"total":96636764160,"available":95117901824,"internally_reserved":0,"allocated":5951488,"data_stored":2970922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.214},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:19.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:20.280 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:20.280 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:20.292 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1636,"stamp":"2023-12-02T22:59:18.893937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847044,"kb_used_data":378760,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268476,"statfs":{"total":289910292480,"available":284946919424,"internally_reserved":0,"allocated":387850240,"data_stored":378788737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-143505,"num_read_kb":-39598511,"num_write":-557162,"num_write_kb":-200975732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9188352,"num_keys_recovered":-9,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002604"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483264,"kb_used_data":5812,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888576,"statfs":{"total":96636764160,"available":95117901824,"internally_reserved":0,"allocated":5951488,"data_stored":2970922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.214},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:20.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:20.569 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:20.570 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:20.582 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1636,"stamp":"2023-12-02T22:59:18.893937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847044,"kb_used_data":378760,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268476,"statfs":{"total":289910292480,"available":284946919424,"internally_reserved":0,"allocated":387850240,"data_stored":378788737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-143505,"num_read_kb":-39598511,"num_write":-557162,"num_write_kb":-200975732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9188352,"num_keys_recovered":-9,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002604"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483264,"kb_used_data":5812,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888576,"statfs":{"total":96636764160,"available":95117901824,"internally_reserved":0,"allocated":5951488,"data_stored":2970922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.214},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:20.583 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:20.583 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:20.583 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:20.583 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:20.583 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:20.583 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:20.584 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:20.584 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:20.584 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:20.584 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:20.584 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:20.584 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 649, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:12.299800+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:12.299800+0000', 'last_peered': '2023-12-02T22:59:12.299800+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:12.299800+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:20.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:20.923 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:20.923 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2286,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963373056,"bytes_avail":284946919424,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:20.936 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:20.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:21.212 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:21.212 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:21.224 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2023-12-02T22:59:20.894269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2160,"num_read_kb":-2046,"num_write":-417,"num_write_kb":-320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.604366"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:21.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-02T22:59:24.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:24.513 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:24.513 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:24.525 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1638,"stamp":"2023-12-02T22:59:22.894653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2160,"num_read_kb":-2046,"num_write":-417,"num_write_kb":-320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002413"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:24.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:24.803 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:24.815 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1638,"stamp":"2023-12-02T22:59:22.894653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2160,"num_read_kb":-2046,"num_write":-417,"num_write_kb":-320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002413"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:24.816 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:24.816 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:24.816 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:24.816 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:24.816 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:24.816 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:24.817 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:24.817 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:24.817 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:24.817 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:24.817 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:24.817 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 649, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:12.299800+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:12.299800+0000', 'last_peered': '2023-12-02T22:59:12.299800+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:12.299800+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:24.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:25.158 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:25.158 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2290,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:25.171 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:25.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:25.445 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:25.445 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:25.457 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2023-12-02T22:59:24.895202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.600306"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:26.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-02T22:59:28.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:28.748 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:28.749 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:28.761 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1640,"stamp":"2023-12-02T22:59:26.895578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:28.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:29.039 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:29.051 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2023-12-02T22:59:28.895997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:29.052 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:29.053 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:29.053 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:29.053 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:29.053 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 649, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:12.299800+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:12.299800+0000', 'last_peered': '2023-12-02T22:59:12.299800+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:12.299800+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:29.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:29.394 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:29.394 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2294,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:29.407 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:29.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:29.682 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:29.682 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:29.694 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2023-12-02T22:59:28.895997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":649,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:12.299800+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:12.299800+0000","last_peered":"2023-12-02T22:59:12.299800+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:12.299800+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:58:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64000000000000001,"15min":0.48199999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.5950000000000002,"5min":9.2420000000000009,"15min":11.510999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.60299999999999998,"15min":0.434},"min":{"1min":0.21199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":1.8819999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:32.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-02T22:59:32.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:32.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:32.986 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:32.999 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2023-12-02T22:59:32.896775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:33.000 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:33.274 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:33.275 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:33.287 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2023-12-02T22:59:32.896775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:33.288 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:33.289 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:33.289 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:33.289 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:33.289 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:33.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:33.628 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:33.628 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2298,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:33.641 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:33.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:33.918 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:33.918 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:33.930 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2023-12-02T22:59:32.896775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:36.932 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:37.220 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:37.221 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:37.233 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2023-12-02T22:59:36.897782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002580"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.42699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:37.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:37.509 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:37.510 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:37.522 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2023-12-02T22:59:36.897782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002580"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.42699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:37.523 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:37.524 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:37.524 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:37.524 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:37.524 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:37.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:37.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-02T22:59:37.865 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:37.865 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2303,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:37.879 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:37.879 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:38.156 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:38.157 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:38.169 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2023-12-02T22:59:36.897782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002580"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.377104+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:13.377104+0000","last_peered":"2023-12-02T22:59:13.377104+0000","last_clean":"2023-12-02T22:59:13.377104+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:13.377104+0000","last_undegraded":"2023-12-02T22:59:13.377104+0000","last_fullsized":"2023-12-02T22:59:13.377104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:58:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.55500000000000005,"15min":0.44600000000000001},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.5910000000000002,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46700000000000003,"15min":0.497},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":5.7089999999999996,"15min":21.295000000000002},"last":0.42699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:41.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:41.459 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:41.459 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:41.472 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2023-12-02T22:59:40.898861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:41.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:41.746 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:41.758 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2023-12-02T22:59:40.898861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:41.759 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:41.759 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:41.759 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:41.759 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:41.759 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:41.759 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:41.760 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:41.760 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:41.760 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:41.760 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:41.760 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:41.760 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:41.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:42.097 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:42.097 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2307,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:42.111 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:42.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:42.388 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:42.388 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:42.401 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2023-12-02T22:59:40.898861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:43.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-02T22:59:45.402 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:45.693 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:45.694 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:45.706 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1649,"stamp":"2023-12-02T22:59:44.899888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:45.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:45.983 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:45.984 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:45.996 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1649,"stamp":"2023-12-02T22:59:44.899888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:45.997 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:45.997 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:45.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:45.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:46.338 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:46.338 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2311,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:46.351 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:46.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:46.626 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:46.627 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:46.639 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1649,"stamp":"2023-12-02T22:59:44.899888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:48.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-02T22:59:49.640 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:49.927 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:49.928 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:49.940 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1651,"stamp":"2023-12-02T22:59:48.900806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:49.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:50.218 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:50.219 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:50.231 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1651,"stamp":"2023-12-02T22:59:48.900806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:50.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:50.232 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:50.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.296640+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:12.296640+0000', 'last_peered': '2023-12-02T22:59:12.296640+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:12.296640+0000', 'last_undegraded': '2023-12-02T22:59:12.296640+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:50.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:50.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:50.573 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:50.574 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2315,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:50.587 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:50.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:50.864 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:50.876 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1651,"stamp":"2023-12-02T22:59:48.900806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.296640+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:12.296640+0000","last_peered":"2023-12-02T22:59:12.296640+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:12.296640+0000","last_undegraded":"2023-12-02T22:59:12.296640+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:53.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:53.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-02T22:59:54.165 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:54.165 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:54.177 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1653,"stamp":"2023-12-02T22:59:52.901679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.30399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.218},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:54.179 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:54.454 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:54.454 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:54.466 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1653,"stamp":"2023-12-02T22:59:52.901679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.30399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.218},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:54.467 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:54.467 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:54.467 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:54.467 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:54.467 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:54.468 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:54.468 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:54.809 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:54.809 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2320,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:54.823 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:54.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:55.099 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:55.099 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:55.111 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2023-12-02T22:59:54.902066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002642"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.38},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.39900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.218},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:58.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:58.399 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:58.400 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:58.412 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2023-12-02T22:59:56.902430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002541"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.38},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.39900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:58.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:58.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:58.693 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:58.705 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2023-12-02T22:59:56.902430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002541"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.38},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.39900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T22:59:58.706 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:58.707 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T22:59:58.707 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T22:59:58.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T22:59:59.051 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:59.051 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2324,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T22:59:59.064 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T22:59:59.064 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T22:59:59.338 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T22:59:59.341 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T22:59:59.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-02T22:59:59.350 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2023-12-02T22:59:58.902831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.38},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.39900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:02.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:02.640 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:02.641 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:02.653 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2023-12-02T23:00:00.903411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002604"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.311},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:02.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:02.931 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:02.931 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:02.943 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2023-12-02T23:00:02.903973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002660"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.311},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:02.944 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:02.944 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:02.945 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:02.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:03.286 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:03.286 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2328,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:03.299 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:03.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:03.574 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:03.574 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:03.586 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2023-12-02T23:00:02.903973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002660"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.311},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:04.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-02T23:00:06.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:06.880 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:06.880 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:06.893 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2023-12-02T23:00:04.904328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002649"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.311},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:06.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:07.172 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:07.184 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2023-12-02T23:00:06.904735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:07.185 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:07.185 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:07.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:07.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:07.526 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:07.527 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2332,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:07.540 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:07.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:07.817 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:07.829 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2023-12-02T23:00:06.904735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:10.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-02T23:00:10.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:11.119 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:11.120 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:11.132 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1662,"stamp":"2023-12-02T23:00:10.905437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:11.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:11.412 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:11.412 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:11.424 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1662,"stamp":"2023-12-02T23:00:10.905437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:11.425 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:11.425 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:11.426 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:11.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:11.765 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:11.766 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2336,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:11.778 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:11.779 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:12.055 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:12.055 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:12.068 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1662,"stamp":"2023-12-02T23:00:10.905437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:15.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:15.358 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:15.358 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:15.370 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1664,"stamp":"2023-12-02T23:00:14.906045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:15.371 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:15.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-02T23:00:15.646 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:15.647 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:15.659 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1664,"stamp":"2023-12-02T23:00:14.906045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:15.660 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:15.660 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:15.661 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:15.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:16.005 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2341,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:16.019 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:16.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:16.294 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:16.295 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:16.307 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1664,"stamp":"2023-12-02T23:00:14.906045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:19.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:19.598 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:19.598 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:19.610 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1666,"stamp":"2023-12-02T23:00:18.906748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.439},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:19.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:19.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:19.886 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:19.898 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1666,"stamp":"2023-12-02T23:00:18.906748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.439},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:19.899 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:19.899 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:19.899 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:19.899 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:19.900 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:19.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:20.241 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:20.241 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2345,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:20.254 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:20.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:20.529 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:20.530 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:20.542 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1666,"stamp":"2023-12-02T23:00:18.906748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.439},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:20.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-02T23:00:23.544 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:23.830 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:23.831 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:23.843 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1668,"stamp":"2023-12-02T23:00:22.907372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001934"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.26400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:23.844 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:24.119 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:24.131 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1668,"stamp":"2023-12-02T23:00:22.907372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001934"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.26400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:24.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:24.132 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:24.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:24.133 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:24.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:24.474 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:24.474 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2349,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:24.487 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:24.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:24.763 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:24.775 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1668,"stamp":"2023-12-02T23:00:22.907372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001934"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.26400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:26.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-02T23:00:27.777 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:28.067 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:28.067 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:28.079 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1670,"stamp":"2023-12-02T23:00:26.908098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.26900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:28.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:28.355 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:28.355 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:28.367 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1670,"stamp":"2023-12-02T23:00:26.908098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.26900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:28.368 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:28.368 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:28.368 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:28.368 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.299643+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:12.299643+0000', 'last_peered': '2023-12-02T22:59:12.299643+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:12.299643+0000', 'last_undegraded': '2023-12-02T22:59:12.299643+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:28.368 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:28.369 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:28.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:28.709 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:28.709 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2353,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:28.722 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:28.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:28.999 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:29.000 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:29.012 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2023-12-02T23:00:28.908681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.299643+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:12.299643+0000","last_peered":"2023-12-02T22:59:12.299643+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:12.299643+0000","last_undegraded":"2023-12-02T22:59:12.299643+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 22:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.56699999999999995,"15min":0.44800000000000001},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.66200000000000003,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.52200000000000002,"15min":0.435},"min":{"1min":0.183,"5min":0.156,"15min":0.156},"max":{"1min":0.42199999999999999,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.26900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:31.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-02T23:00:32.014 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:32.300 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:32.301 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:32.314 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2023-12-02T23:00:30.909041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002293"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:32.315 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:32.590 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:32.590 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:32.602 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2023-12-02T23:00:30.909041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002293"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:32.603 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:32.603 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:32.603 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.494185+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:14.494185+0000', 'last_peered': '2023-12-02T22:59:14.494185+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:14.494185+0000', 'last_undegraded': '2023-12-02T22:59:14.494185+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:32.604 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:32.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:32.944 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:32.945 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2358,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:32.957 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:32.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:33.233 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:33.234 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:33.246 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2023-12-02T23:00:32.909634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002568"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:36.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:36.534 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:36.534 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:36.546 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2023-12-02T23:00:34.910043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:36.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:36.823 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:36.823 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:36.835 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2023-12-02T23:00:34.910043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.297892+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:12.297892+0000","last_peered":"2023-12-02T22:59:12.297892+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:12.297892+0000","last_undegraded":"2023-12-02T22:59:12.297892+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 518, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.297892+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:12.297892+0000', 'last_peered': '2023-12-02T22:59:12.297892+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:12.297892+0000', 'last_undegraded': '2023-12-02T22:59:12.297892+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.494185+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:14.494185+0000', 'last_peered': '2023-12-02T22:59:14.494185+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:14.494185+0000', 'last_undegraded': '2023-12-02T22:59:14.494185+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:36.836 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:36.837 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:36.837 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:36.837 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:36.837 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:36.837 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:37.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-02T23:00:37.177 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:37.177 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2362,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:37.190 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:37.191 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:37.466 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:37.466 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:37.479 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1675,"stamp":"2023-12-02T23:00:36.910454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494693+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:14.494693+0000","last_peered":"2023-12-02T22:59:14.494693+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:14.494693+0000","last_undegraded":"2023-12-02T22:59:14.494693+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.224},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.20499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:40.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:40.767 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:40.767 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:40.779 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2023-12-02T23:00:38.910824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494693+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:14.494693+0000","last_peered":"2023-12-02T22:59:14.494693+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:14.494693+0000","last_undegraded":"2023-12-02T22:59:14.494693+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.224},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.20499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:40.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:41.055 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:41.067 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1677,"stamp":"2023-12-02T23:00:40.911174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002492"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494693+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:14.494693+0000","last_peered":"2023-12-02T22:59:14.494693+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:14.494693+0000","last_undegraded":"2023-12-02T22:59:14.494693+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.24399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:41.068 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-02T23:00:41.068 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.494693+0000', 'last_change': '2023-12-02T22:55:38.896684+0000', 'last_active': '2023-12-02T22:59:14.494693+0000', 'last_peered': '2023-12-02T22:59:14.494693+0000', 'last_clean': '2023-12-02T22:55:35.288022+0000', 'last_became_active': '2023-12-02T22:55:38.896684+0000', 'last_became_peered': '2023-12-02T22:55:38.896684+0000', 'last_unstale': '2023-12-02T22:59:14.494693+0000', 'last_undegraded': '2023-12-02T22:59:14.494693+0000', 'last_fullsized': '2023-12-02T22:55:38.880192+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:41.068 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-02T23:00:41.068 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.494185+0000', 'last_change': '2023-12-02T22:55:38.896378+0000', 'last_active': '2023-12-02T22:59:14.494185+0000', 'last_peered': '2023-12-02T22:59:14.494185+0000', 'last_clean': '2023-12-02T22:55:35.288434+0000', 'last_became_active': '2023-12-02T22:55:38.896378+0000', 'last_became_peered': '2023-12-02T22:55:38.896378+0000', 'last_unstale': '2023-12-02T22:59:14.494185+0000', 'last_undegraded': '2023-12-02T22:59:14.494185+0000', 'last_fullsized': '2023-12-02T22:55:38.879436+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:41.068 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298378+0000', 'last_change': '2023-12-02T22:55:38.896027+0000', 'last_active': '2023-12-02T22:59:12.298378+0000', 'last_peered': '2023-12-02T22:59:12.298378+0000', 'last_clean': '2023-12-02T22:55:35.287966+0000', 'last_became_active': '2023-12-02T22:55:38.896027+0000', 'last_became_peered': '2023-12-02T22:55:38.896027+0000', 'last_unstale': '2023-12-02T22:59:12.298378+0000', 'last_undegraded': '2023-12-02T22:59:12.298378+0000', 'last_fullsized': '2023-12-02T22:55:38.877779+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:12.298918+0000', 'last_change': '2023-12-02T22:55:38.882136+0000', 'last_active': '2023-12-02T22:59:12.298918+0000', 'last_peered': '2023-12-02T22:59:12.298918+0000', 'last_clean': '2023-12-02T22:55:35.287406+0000', 'last_became_active': '2023-12-02T22:55:38.882136+0000', 'last_became_peered': '2023-12-02T22:55:38.882136+0000', 'last_unstale': '2023-12-02T22:59:12.298918+0000', 'last_undegraded': '2023-12-02T22:59:12.298918+0000', 'last_fullsized': '2023-12-02T22:55:38.874302+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 501, 'state': 'active+undersized', 'last_fresh': '2023-12-02T22:59:14.493830+0000', 'last_change': '2023-12-02T22:55:38.898825+0000', 'last_active': '2023-12-02T22:59:14.493830+0000', 'last_peered': '2023-12-02T22:59:14.493830+0000', 'last_clean': '2023-12-02T22:21:30.779823+0000', 'last_became_active': '2023-12-02T22:55:38.898825+0000', 'last_became_peered': '2023-12-02T22:55:38.898825+0000', 'last_unstale': '2023-12-02T22:59:14.493830+0000', 'last_undegraded': '2023-12-02T22:59:14.493830+0000', 'last_fullsized': '2023-12-02T22:55:38.875392+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:30.779823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-02T23:00:41.069 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 650, 'reported_epoch': 501, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-02T22:59:14.491540+0000', 'last_change': '2023-12-02T22:55:38.889145+0000', 'last_active': '2023-12-02T22:59:14.491540+0000', 'last_peered': '2023-12-02T22:59:14.491540+0000', 'last_clean': '2023-12-02T22:55:35.284282+0000', 'last_became_active': '2023-12-02T22:55:38.889145+0000', 'last_became_peered': '2023-12-02T22:55:38.889145+0000', 'last_unstale': '2023-12-02T22:59:14.491540+0000', 'last_undegraded': '2023-12-02T22:55:38.879064+0000', 'last_fullsized': '2023-12-02T22:55:38.875176+0000', 'mapping_epoch': 484, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'last_clean_scrub_stamp': '2023-12-02T22:21:24.514659+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-02T23:00:41.069 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-02T23:00:41.408 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:41.408 INFO:teuthology.orchestra.run.smithi170.stdout:{"fsid":"5e1eb4f6-095d-4fc4-8d62-43f3f257587e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2366,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":501,"num_osds":3,"num_up_osds":2,"osd_up_since":1701557737,"num_in_osds":3,"osd_in_since":1701555675,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4963385344,"bytes_avail":284946907136,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":46,"modified":"2023-12-02T22:59:16.893395+0000","services":{}},"progress_events":{}} 2023-12-02T23:00:41.421 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-02T23:00:41.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:00:41.696 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-02T23:00:41.696 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-12-02T23:00:41.708 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1677,"stamp":"2023-12-02T23:00:40.911174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4847056,"kb_used_data":378772,"kb_used_omap":276,"kb_used_meta":4467499,"kb_avail":278268464,"statfs":{"total":289910292480,"available":284946907136,"internally_reserved":0,"allocated":387862528,"data_stored":378795450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":283581,"internal_metadata":4574719043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002492"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494693+0000","last_change":"2023-12-02T22:55:38.896684+0000","last_active":"2023-12-02T22:59:14.494693+0000","last_peered":"2023-12-02T22:59:14.494693+0000","last_clean":"2023-12-02T22:55:35.288022+0000","last_became_active":"2023-12-02T22:55:38.896684+0000","last_became_peered":"2023-12-02T22:55:38.896684+0000","last_unstale":"2023-12-02T22:59:14.494693+0000","last_undegraded":"2023-12-02T22:59:14.494693+0000","last_fullsized":"2023-12-02T22:55:38.880192+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:36:54.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.494185+0000","last_change":"2023-12-02T22:55:38.896378+0000","last_active":"2023-12-02T22:59:14.494185+0000","last_peered":"2023-12-02T22:59:14.494185+0000","last_clean":"2023-12-02T22:55:35.288434+0000","last_became_active":"2023-12-02T22:55:38.896378+0000","last_became_peered":"2023-12-02T22:55:38.896378+0000","last_unstale":"2023-12-02T22:59:14.494185+0000","last_undegraded":"2023-12-02T22:59:14.494185+0000","last_fullsized":"2023-12-02T22:55:38.879436+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:51:15.495155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298378+0000","last_change":"2023-12-02T22:55:38.896027+0000","last_active":"2023-12-02T22:59:12.298378+0000","last_peered":"2023-12-02T22:59:12.298378+0000","last_clean":"2023-12-02T22:55:35.287966+0000","last_became_active":"2023-12-02T22:55:38.896027+0000","last_became_peered":"2023-12-02T22:55:38.896027+0000","last_unstale":"2023-12-02T22:59:12.298378+0000","last_undegraded":"2023-12-02T22:59:12.298378+0000","last_fullsized":"2023-12-02T22:55:38.877779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:19:38.464114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-12-02T22:59:12.298918+0000","last_change":"2023-12-02T22:55:38.882136+0000","last_active":"2023-12-02T22:59:12.298918+0000","last_peered":"2023-12-02T22:59:12.298918+0000","last_clean":"2023-12-02T22:55:35.287406+0000","last_became_active":"2023-12-02T22:55:38.882136+0000","last_became_peered":"2023-12-02T22:55:38.882136+0000","last_unstale":"2023-12-02T22:59:12.298918+0000","last_undegraded":"2023-12-02T22:59:12.298918+0000","last_fullsized":"2023-12-02T22:55:38.874302+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:55:56.824316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":501,"state":"active+undersized","last_fresh":"2023-12-02T22:59:14.493830+0000","last_change":"2023-12-02T22:55:38.898825+0000","last_active":"2023-12-02T22:59:14.493830+0000","last_peered":"2023-12-02T22:59:14.493830+0000","last_clean":"2023-12-02T22:21:30.779823+0000","last_became_active":"2023-12-02T22:55:38.898825+0000","last_became_peered":"2023-12-02T22:55:38.898825+0000","last_unstale":"2023-12-02T22:59:14.493830+0000","last_undegraded":"2023-12-02T22:59:14.493830+0000","last_fullsized":"2023-12-02T22:55:38.875392+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:13:50.136675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.373533+0000","last_change":"2023-12-02T22:21:31.792676+0000","last_active":"2023-12-02T22:59:13.373533+0000","last_peered":"2023-12-02T22:59:13.373533+0000","last_clean":"2023-12-02T22:59:13.373533+0000","last_became_active":"2023-12-02T22:21:31.792357+0000","last_became_peered":"2023-12-02T22:21:31.792357+0000","last_unstale":"2023-12-02T22:59:13.373533+0000","last_undegraded":"2023-12-02T22:59:13.373533+0000","last_fullsized":"2023-12-02T22:59:13.373533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:48:45.269877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":501,"state":"active+clean","last_fresh":"2023-12-02T22:59:14.297047+0000","last_change":"2023-12-02T22:21:31.791125+0000","last_active":"2023-12-02T22:59:14.297047+0000","last_peered":"2023-12-02T22:59:14.297047+0000","last_clean":"2023-12-02T22:59:14.297047+0000","last_became_active":"2023-12-02T22:21:31.790644+0000","last_became_peered":"2023-12-02T22:21:31.790644+0000","last_unstale":"2023-12-02T22:59:14.297047+0000","last_undegraded":"2023-12-02T22:59:14.297047+0000","last_fullsized":"2023-12-02T22:59:14.297047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T06:44:09.849606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-12-02T22:59:12.298206+0000","last_change":"2023-12-02T22:21:31.790486+0000","last_active":"2023-12-02T22:59:12.298206+0000","last_peered":"2023-12-02T22:59:12.298206+0000","last_clean":"2023-12-02T22:59:12.298206+0000","last_became_active":"2023-12-02T22:21:31.790162+0000","last_became_peered":"2023-12-02T22:21:31.790162+0000","last_unstale":"2023-12-02T22:59:12.298206+0000","last_undegraded":"2023-12-02T22:59:12.298206+0000","last_fullsized":"2023-12-02T22:59:12.298206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:30.779823+0000","last_clean_scrub_stamp":"2023-12-02T22:21:30.779823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:39:51.574788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":650,"reported_epoch":501,"state":"active+undersized+degraded","last_fresh":"2023-12-02T22:59:14.491540+0000","last_change":"2023-12-02T22:55:38.889145+0000","last_active":"2023-12-02T22:59:14.491540+0000","last_peered":"2023-12-02T22:59:14.491540+0000","last_clean":"2023-12-02T22:55:35.284282+0000","last_became_active":"2023-12-02T22:55:38.889145+0000","last_became_peered":"2023-12-02T22:55:38.889145+0000","last_unstale":"2023-12-02T22:59:14.491540+0000","last_undegraded":"2023-12-02T22:55:38.879064+0000","last_fullsized":"2023-12-02T22:55:38.875176+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:21:24.514659+0000","last_clean_scrub_stamp":"2023-12-02T22:21:24.514659+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:49:03.051143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1483276,"kb_used_data":5824,"kb_used_omap":83,"kb_used_meta":1477356,"kb_avail":92888564,"statfs":{"total":96636764160,"available":95117889536,"internally_reserved":0,"allocated":5963776,"data_stored":2977635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85676,"internal_metadata":1512812884},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:55:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.61199999999999999,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":6.883,"5min":7.3550000000000004,"15min":7.3550000000000004},"last":0.251},{"interface":"front","average":{"1min":0.871,"5min":0.51100000000000001,"15min":0.48799999999999999},"min":{"1min":0.26200000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":12.226000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.20499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41699999999999998,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.38100000000000001,"15min":0.435},"min":{"1min":0.23100000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.442,"5min":1.8819999999999999,"15min":8.2469999999999999},"last":0.24399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1489872,"kb_used_data":6404,"kb_used_omap":92,"kb_used_meta":1483363,"kb_avail":92881968,"statfs":{"total":96636764160,"available":95111135232,"internally_reserved":0,"allocated":6557696,"data_stored":3568003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94990,"internal_metadata":1518963954},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 22:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.94199999999999995,"15min":0.66100000000000003},"min":{"1min":0.23300000000000001,"5min":0.156,"15min":0.14199999999999999},"max":{"1min":5.6600000000000001,"5min":37.250999999999998,"15min":37.250999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.45000000000000001,"15min":0.39500000000000002},"min":{"1min":0.217,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.847,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Dec 2 22:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.48499999999999999,"15min":0.441},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.40899999999999997,"5min":5.8520000000000003,"15min":5.8520000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.38600000000000001,"15min":0.49099999999999999},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":0.42699999999999999,"5min":2.9910000000000001,"15min":21.295000000000002},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1873908,"kb_used_data":366544,"kb_used_omap":100,"kb_used_meta":1506779,"kb_avail":92497932,"statfs":{"total":96636764160,"available":94717882368,"internally_reserved":0,"allocated":375341056,"data_stored":372249812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102915,"internal_metadata":1542942205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.14899999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":1.3919999999999999,"5min":2.3900000000000001,"15min":5.1230000000000002},"last":17.225999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":1.3500000000000001,"5min":3.2629999999999999,"15min":10.087},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Dec 2 22:55:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.53500000000000003,"15min":0.44700000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.71399999999999997,"5min":7.4779999999999998,"15min":10.135999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.56999999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":0.82999999999999996,"5min":7.4409999999999998,"15min":7.4409999999999998},"last":3.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:00:42.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-02T23:00:42.576 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-02T23:00:42.979 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-02T23:00:42.979 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-02T23:00:42.980 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-02T23:00:42.980 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-02T23:00:42.981 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f4a359d5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 85862) UID: 0 2023-12-02T23:00:42.981 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f4a359d5640 -1 osd.1 501 *** Got signal Terminated *** 2023-12-02T23:00:42.981 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f4a359d5640 -1 osd.1 501 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T23:00:42.981 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f2e2f9cc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 85863) UID: 0 2023-12-02T23:00:42.981 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f2e2f9cc640 -1 osd.2 501 *** Got signal Terminated *** 2023-12-02T23:00:42.981 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f2e2f9cc640 -1 osd.2 501 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T23:00:42.982 INFO:tasks.ceph.cluster2.mon.a.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f7fcea10640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 85544) UID: 0 2023-12-02T23:00:42.982 INFO:tasks.ceph.cluster2.mon.a.smithi170.stderr:2023-12-02T23:00:42.979+0000 7f7fcea10640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-02T23:00:42.983 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T23:00:42.979+0000 7fa5c5793640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 90513) UID: 0 2023-12-02T23:00:42.983 INFO:tasks.ceph.cluster2.mgr.x.smithi012.stderr:2023-12-02T23:00:42.979+0000 7fa5c5793640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-02T23:00:44.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T23:02:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-02T23:02:44.750 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T23:02:44.752 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-02T23:02:44.752 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-02T23:02:44.753 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-02T23:02:44.753 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:02:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-02T23:02:44.753 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-02T23:02:44.754 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-02T23:02:44.754 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-02T23:02:44.754 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:02:44.754 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-02T23:02:44.754 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-02T23:02:44.754 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:02:44.754 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-02T23:02:44.754 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-02T23:02:44.755 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-02T23:02:44.755 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:02:44.755 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-02T23:02:44.755 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-02T23:02:44.755 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-02T23:02:44.755 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:02:44.755 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-02T23:02:44.755 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-02T23:02:44.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-02T23:02:44.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T23:02:44.823 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-02T23:02:45.032 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T23:02:45.032 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-02T23:02:45.235 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T23:02:45.235 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-02T23:02:45.414 INFO:tasks.ceph:Archiving mon data... 2023-12-02T23:02:46.123 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/data/cluster2.mon.a.tgz 2023-12-02T23:02:46.124 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-12-02T23:02:46.151 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.eXoyqfkEE9 2023-12-02T23:02:46.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.eXoyqfkEE9 2023-12-02T23:02:46.331 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.eXoyqfkEE9 2023-12-02T23:02:46.437 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.eXoyqfkEE9 is 437KB 2023-12-02T23:02:46.534 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.eXoyqfkEE9 2023-12-02T23:02:46.550 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-02T23:02:46.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T23:02:46.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T23:02:46.629 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T23:02:46.630 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-02T23:02:46.645 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-02T23:02:46.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T23:02:46.946 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T23:02:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T23:02:46.958 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1718,"stamp":"2023-12-02T23:02:46.592134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":448532,"kb_used_data":12272,"kb_used_omap":4,"kb_used_meta":436219,"kb_avail":282666988,"statfs":{"total":289910292480,"available":289450995712,"internally_reserved":0,"allocated":12566528,"data_stored":7294433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":446688591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002302"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:59.805595+0000","last_change":"2023-12-02T22:20:56.564305+0000","last_active":"2023-12-02T22:59:59.805595+0000","last_peered":"2023-12-02T22:59:59.805595+0000","last_clean":"2023-12-02T22:59:59.805595+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T22:59:59.805595+0000","last_undegraded":"2023-12-02T22:59:59.805595+0000","last_fullsized":"2023-12-02T22:59:59.805595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:31:54.028189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:54.805010+0000","last_change":"2023-12-02T22:20:56.565011+0000","last_active":"2023-12-02T22:59:54.805010+0000","last_peered":"2023-12-02T22:59:54.805010+0000","last_clean":"2023-12-02T22:59:54.805010+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T22:59:54.805010+0000","last_undegraded":"2023-12-02T22:59:54.805010+0000","last_fullsized":"2023-12-02T22:59:54.805010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:00:19.409954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:39.809627+0000","last_change":"2023-12-02T22:20:56.563555+0000","last_active":"2023-12-02T23:00:39.809627+0000","last_peered":"2023-12-02T23:00:39.809627+0000","last_clean":"2023-12-02T23:00:39.809627+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T23:00:39.809627+0000","last_undegraded":"2023-12-02T23:00:39.809627+0000","last_fullsized":"2023-12-02T23:00:39.809627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T09:35:24.530609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809158+0000","last_change":"2023-12-02T22:20:56.562760+0000","last_active":"2023-12-02T23:00:34.809158+0000","last_peered":"2023-12-02T23:00:34.809158+0000","last_clean":"2023-12-02T23:00:34.809158+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T23:00:34.809158+0000","last_undegraded":"2023-12-02T23:00:34.809158+0000","last_fullsized":"2023-12-02T23:00:34.809158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:42:42.364951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.815720+0000","last_change":"2023-12-02T22:20:56.562058+0000","last_active":"2023-12-02T23:00:34.815720+0000","last_peered":"2023-12-02T23:00:34.815720+0000","last_clean":"2023-12-02T23:00:34.815720+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T23:00:34.815720+0000","last_undegraded":"2023-12-02T23:00:34.815720+0000","last_fullsized":"2023-12-02T23:00:34.815720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:09:32.035065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:29.788062+0000","last_change":"2023-12-02T22:20:56.563338+0000","last_active":"2023-12-02T23:00:29.788062+0000","last_peered":"2023-12-02T23:00:29.788062+0000","last_clean":"2023-12-02T23:00:29.788062+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T23:00:29.788062+0000","last_undegraded":"2023-12-02T23:00:29.788062+0000","last_fullsized":"2023-12-02T23:00:29.788062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T23:03:15.596248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.807941+0000","last_change":"2023-12-02T22:20:56.565388+0000","last_active":"2023-12-02T22:59:13.807941+0000","last_peered":"2023-12-02T22:59:13.807941+0000","last_clean":"2023-12-02T22:59:13.807941+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T22:59:13.807941+0000","last_undegraded":"2023-12-02T22:59:13.807941+0000","last_fullsized":"2023-12-02T22:59:13.807941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:24:24.875735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809141+0000","last_change":"2023-12-02T22:20:56.563039+0000","last_active":"2023-12-02T23:00:34.809141+0000","last_peered":"2023-12-02T23:00:34.809141+0000","last_clean":"2023-12-02T23:00:34.809141+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T23:00:34.809141+0000","last_undegraded":"2023-12-02T23:00:34.809141+0000","last_fullsized":"2023-12-02T23:00:34.809141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:37:53.638783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":608,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.101290+0000","last_change":"2023-12-02T22:20:51.698937+0000","last_active":"2023-12-02T22:59:13.101290+0000","last_peered":"2023-12-02T22:59:13.101290+0000","last_clean":"2023-12-02T22:59:13.101290+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T22:59:13.101290+0000","last_undegraded":"2023-12-02T22:59:13.101290+0000","last_fullsized":"2023-12-02T22:59:13.101290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_clean_scrub_stamp":"2023-12-02T22:20:50.530145+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:30:49.984827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138948,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":94232892,"statfs":{"total":96636764160,"available":96494481408,"internally_reserved":0,"allocated":3792896,"data_stored":2037899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32100000000000001,"15min":0.30599999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":0.441,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.308},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.216,"5min":0.156,"15min":0.156},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.502},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.23899999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41799999999999998,"5min":0.54700000000000004,"15min":0.83699999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.95699999999999996},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129800,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242040,"statfs":{"total":96636764160,"available":96503848960,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32600000000000001,"15min":0.32800000000000001},"min":{"1min":0.224,"5min":0.20699999999999999,"15min":0.16600000000000001},"max":{"1min":0.42799999999999999,"5min":0.46899999999999997,"15min":2.9550000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.48799999999999999,"15min":0.54900000000000004},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34100000000000003,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19400000000000001},"max":{"1min":0.45400000000000001,"5min":0.51600000000000001,"15min":1.129},"last":0.433},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.19500000000000001},"max":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":2.4169999999999998},"last":0.46700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179784,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192056,"statfs":{"total":96636764160,"available":96452665344,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33100000000000002,"15min":0.317},"min":{"1min":0.219,"5min":0.219,"15min":0.16},"max":{"1min":0.504,"5min":0.51000000000000001,"15min":1.5549999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.34300000000000003},"min":{"1min":0.217,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":0.504,"5min":0.53700000000000003,"15min":0.67800000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":2.0070000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.37},"min":{"1min":0.32900000000000001,"5min":0.26100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:02:46.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T23:02:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T23:02:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T23:02:47.272 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1718,"stamp":"2023-12-02T23:02:46.592134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":448532,"kb_used_data":12272,"kb_used_omap":4,"kb_used_meta":436219,"kb_avail":282666988,"statfs":{"total":289910292480,"available":289450995712,"internally_reserved":0,"allocated":12566528,"data_stored":7294433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":446688591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002302"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:59.805595+0000","last_change":"2023-12-02T22:20:56.564305+0000","last_active":"2023-12-02T22:59:59.805595+0000","last_peered":"2023-12-02T22:59:59.805595+0000","last_clean":"2023-12-02T22:59:59.805595+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T22:59:59.805595+0000","last_undegraded":"2023-12-02T22:59:59.805595+0000","last_fullsized":"2023-12-02T22:59:59.805595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:31:54.028189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:54.805010+0000","last_change":"2023-12-02T22:20:56.565011+0000","last_active":"2023-12-02T22:59:54.805010+0000","last_peered":"2023-12-02T22:59:54.805010+0000","last_clean":"2023-12-02T22:59:54.805010+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T22:59:54.805010+0000","last_undegraded":"2023-12-02T22:59:54.805010+0000","last_fullsized":"2023-12-02T22:59:54.805010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:00:19.409954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:39.809627+0000","last_change":"2023-12-02T22:20:56.563555+0000","last_active":"2023-12-02T23:00:39.809627+0000","last_peered":"2023-12-02T23:00:39.809627+0000","last_clean":"2023-12-02T23:00:39.809627+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T23:00:39.809627+0000","last_undegraded":"2023-12-02T23:00:39.809627+0000","last_fullsized":"2023-12-02T23:00:39.809627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T09:35:24.530609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809158+0000","last_change":"2023-12-02T22:20:56.562760+0000","last_active":"2023-12-02T23:00:34.809158+0000","last_peered":"2023-12-02T23:00:34.809158+0000","last_clean":"2023-12-02T23:00:34.809158+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T23:00:34.809158+0000","last_undegraded":"2023-12-02T23:00:34.809158+0000","last_fullsized":"2023-12-02T23:00:34.809158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:42:42.364951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.815720+0000","last_change":"2023-12-02T22:20:56.562058+0000","last_active":"2023-12-02T23:00:34.815720+0000","last_peered":"2023-12-02T23:00:34.815720+0000","last_clean":"2023-12-02T23:00:34.815720+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T23:00:34.815720+0000","last_undegraded":"2023-12-02T23:00:34.815720+0000","last_fullsized":"2023-12-02T23:00:34.815720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:09:32.035065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:29.788062+0000","last_change":"2023-12-02T22:20:56.563338+0000","last_active":"2023-12-02T23:00:29.788062+0000","last_peered":"2023-12-02T23:00:29.788062+0000","last_clean":"2023-12-02T23:00:29.788062+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T23:00:29.788062+0000","last_undegraded":"2023-12-02T23:00:29.788062+0000","last_fullsized":"2023-12-02T23:00:29.788062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T23:03:15.596248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.807941+0000","last_change":"2023-12-02T22:20:56.565388+0000","last_active":"2023-12-02T22:59:13.807941+0000","last_peered":"2023-12-02T22:59:13.807941+0000","last_clean":"2023-12-02T22:59:13.807941+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T22:59:13.807941+0000","last_undegraded":"2023-12-02T22:59:13.807941+0000","last_fullsized":"2023-12-02T22:59:13.807941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:24:24.875735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809141+0000","last_change":"2023-12-02T22:20:56.563039+0000","last_active":"2023-12-02T23:00:34.809141+0000","last_peered":"2023-12-02T23:00:34.809141+0000","last_clean":"2023-12-02T23:00:34.809141+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T23:00:34.809141+0000","last_undegraded":"2023-12-02T23:00:34.809141+0000","last_fullsized":"2023-12-02T23:00:34.809141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:37:53.638783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":608,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.101290+0000","last_change":"2023-12-02T22:20:51.698937+0000","last_active":"2023-12-02T22:59:13.101290+0000","last_peered":"2023-12-02T22:59:13.101290+0000","last_clean":"2023-12-02T22:59:13.101290+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T22:59:13.101290+0000","last_undegraded":"2023-12-02T22:59:13.101290+0000","last_fullsized":"2023-12-02T22:59:13.101290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_clean_scrub_stamp":"2023-12-02T22:20:50.530145+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:30:49.984827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138948,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":94232892,"statfs":{"total":96636764160,"available":96494481408,"internally_reserved":0,"allocated":3792896,"data_stored":2037899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32100000000000001,"15min":0.30599999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":0.441,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.308},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.216,"5min":0.156,"15min":0.156},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.502},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.23899999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41799999999999998,"5min":0.54700000000000004,"15min":0.83699999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.95699999999999996},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129800,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242040,"statfs":{"total":96636764160,"available":96503848960,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32600000000000001,"15min":0.32800000000000001},"min":{"1min":0.224,"5min":0.20699999999999999,"15min":0.16600000000000001},"max":{"1min":0.42799999999999999,"5min":0.46899999999999997,"15min":2.9550000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.48799999999999999,"15min":0.54900000000000004},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34100000000000003,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19400000000000001},"max":{"1min":0.45400000000000001,"5min":0.51600000000000001,"15min":1.129},"last":0.433},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.19500000000000001},"max":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":2.4169999999999998},"last":0.46700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179784,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192056,"statfs":{"total":96636764160,"available":96452665344,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33100000000000002,"15min":0.317},"min":{"1min":0.219,"5min":0.219,"15min":0.16},"max":{"1min":0.504,"5min":0.51000000000000001,"15min":1.5549999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.34300000000000003},"min":{"1min":0.217,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":0.504,"5min":0.53700000000000003,"15min":0.67800000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":2.0070000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.37},"min":{"1min":0.32900000000000001,"5min":0.26100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:02:47.273 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-02T23:02:47.273 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T23:02:47.570 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T23:02:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T23:02:47.583 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1718,"stamp":"2023-12-02T23:02:46.592134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":448532,"kb_used_data":12272,"kb_used_omap":4,"kb_used_meta":436219,"kb_avail":282666988,"statfs":{"total":289910292480,"available":289450995712,"internally_reserved":0,"allocated":12566528,"data_stored":7294433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":446688591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002302"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:59.805595+0000","last_change":"2023-12-02T22:20:56.564305+0000","last_active":"2023-12-02T22:59:59.805595+0000","last_peered":"2023-12-02T22:59:59.805595+0000","last_clean":"2023-12-02T22:59:59.805595+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T22:59:59.805595+0000","last_undegraded":"2023-12-02T22:59:59.805595+0000","last_fullsized":"2023-12-02T22:59:59.805595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:31:54.028189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:54.805010+0000","last_change":"2023-12-02T22:20:56.565011+0000","last_active":"2023-12-02T22:59:54.805010+0000","last_peered":"2023-12-02T22:59:54.805010+0000","last_clean":"2023-12-02T22:59:54.805010+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T22:59:54.805010+0000","last_undegraded":"2023-12-02T22:59:54.805010+0000","last_fullsized":"2023-12-02T22:59:54.805010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:00:19.409954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:39.809627+0000","last_change":"2023-12-02T22:20:56.563555+0000","last_active":"2023-12-02T23:00:39.809627+0000","last_peered":"2023-12-02T23:00:39.809627+0000","last_clean":"2023-12-02T23:00:39.809627+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T23:00:39.809627+0000","last_undegraded":"2023-12-02T23:00:39.809627+0000","last_fullsized":"2023-12-02T23:00:39.809627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T09:35:24.530609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809158+0000","last_change":"2023-12-02T22:20:56.562760+0000","last_active":"2023-12-02T23:00:34.809158+0000","last_peered":"2023-12-02T23:00:34.809158+0000","last_clean":"2023-12-02T23:00:34.809158+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T23:00:34.809158+0000","last_undegraded":"2023-12-02T23:00:34.809158+0000","last_fullsized":"2023-12-02T23:00:34.809158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:42:42.364951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.815720+0000","last_change":"2023-12-02T22:20:56.562058+0000","last_active":"2023-12-02T23:00:34.815720+0000","last_peered":"2023-12-02T23:00:34.815720+0000","last_clean":"2023-12-02T23:00:34.815720+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T23:00:34.815720+0000","last_undegraded":"2023-12-02T23:00:34.815720+0000","last_fullsized":"2023-12-02T23:00:34.815720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:09:32.035065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:29.788062+0000","last_change":"2023-12-02T22:20:56.563338+0000","last_active":"2023-12-02T23:00:29.788062+0000","last_peered":"2023-12-02T23:00:29.788062+0000","last_clean":"2023-12-02T23:00:29.788062+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T23:00:29.788062+0000","last_undegraded":"2023-12-02T23:00:29.788062+0000","last_fullsized":"2023-12-02T23:00:29.788062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T23:03:15.596248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.807941+0000","last_change":"2023-12-02T22:20:56.565388+0000","last_active":"2023-12-02T22:59:13.807941+0000","last_peered":"2023-12-02T22:59:13.807941+0000","last_clean":"2023-12-02T22:59:13.807941+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T22:59:13.807941+0000","last_undegraded":"2023-12-02T22:59:13.807941+0000","last_fullsized":"2023-12-02T22:59:13.807941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:24:24.875735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809141+0000","last_change":"2023-12-02T22:20:56.563039+0000","last_active":"2023-12-02T23:00:34.809141+0000","last_peered":"2023-12-02T23:00:34.809141+0000","last_clean":"2023-12-02T23:00:34.809141+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T23:00:34.809141+0000","last_undegraded":"2023-12-02T23:00:34.809141+0000","last_fullsized":"2023-12-02T23:00:34.809141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:37:53.638783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":608,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.101290+0000","last_change":"2023-12-02T22:20:51.698937+0000","last_active":"2023-12-02T22:59:13.101290+0000","last_peered":"2023-12-02T22:59:13.101290+0000","last_clean":"2023-12-02T22:59:13.101290+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T22:59:13.101290+0000","last_undegraded":"2023-12-02T22:59:13.101290+0000","last_fullsized":"2023-12-02T22:59:13.101290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_clean_scrub_stamp":"2023-12-02T22:20:50.530145+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:30:49.984827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138948,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":94232892,"statfs":{"total":96636764160,"available":96494481408,"internally_reserved":0,"allocated":3792896,"data_stored":2037899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32100000000000001,"15min":0.30599999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":0.441,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.308},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.216,"5min":0.156,"15min":0.156},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.502},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.23899999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41799999999999998,"5min":0.54700000000000004,"15min":0.83699999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.95699999999999996},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129800,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242040,"statfs":{"total":96636764160,"available":96503848960,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32600000000000001,"15min":0.32800000000000001},"min":{"1min":0.224,"5min":0.20699999999999999,"15min":0.16600000000000001},"max":{"1min":0.42799999999999999,"5min":0.46899999999999997,"15min":2.9550000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.48799999999999999,"15min":0.54900000000000004},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34100000000000003,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19400000000000001},"max":{"1min":0.45400000000000001,"5min":0.51600000000000001,"15min":1.129},"last":0.433},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.19500000000000001},"max":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":2.4169999999999998},"last":0.46700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179784,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192056,"statfs":{"total":96636764160,"available":96452665344,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33100000000000002,"15min":0.317},"min":{"1min":0.219,"5min":0.219,"15min":0.16},"max":{"1min":0.504,"5min":0.51000000000000001,"15min":1.5549999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.34300000000000003},"min":{"1min":0.217,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":0.504,"5min":0.53700000000000003,"15min":0.67800000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":2.0070000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.37},"min":{"1min":0.32900000000000001,"5min":0.26100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:02:47.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T23:02:47.879 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T23:02:47.879 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":461,"fsid":"1b87a8aa-c41b-4602-a35b-641c227d62c2","created":"2023-12-02T22:20:39.540431+0000","modified":"2023-12-02T22:59:13.092686+0000","last_up_change":"2023-12-02T22:20:49.358782+0000","last_in_change":"2023-12-02T22:20:40.442626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T22:20:50.019027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T22:20:54.851652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04d4736-d721-4737-913f-88be3aed59dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6801","nonce":3782931675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6803","nonce":3782931675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6807","nonce":3782931675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3782931675},{"type":"v1","addr":"172.21.15.12:6805","nonce":3782931675}]},"public_addr":"172.21.15.12:6801/3782931675","cluster_addr":"172.21.15.12:6803/3782931675","heartbeat_back_addr":"172.21.15.12:6807/3782931675","heartbeat_front_addr":"172.21.15.12:6805/3782931675","state":["exists","up"]},{"osd":1,"uuid":"816087c7-a78c-48b7-9ad4-b24ff37a888a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6809","nonce":3639128561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6811","nonce":3639128561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6815","nonce":3639128561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3639128561},{"type":"v1","addr":"172.21.15.12:6813","nonce":3639128561}]},"public_addr":"172.21.15.12:6809/3639128561","cluster_addr":"172.21.15.12:6811/3639128561","heartbeat_back_addr":"172.21.15.12:6815/3639128561","heartbeat_front_addr":"172.21.15.12:6813/3639128561","state":["exists","up"]},{"osd":2,"uuid":"e9bdc8c4-9464-4db4-98b0-82dd4154703d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6817","nonce":4018360220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6819","nonce":4018360220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6823","nonce":4018360220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4018360220},{"type":"v1","addr":"172.21.15.12:6821","nonce":4018360220}]},"public_addr":"172.21.15.12:6817/4018360220","cluster_addr":"172.21.15.12:6819/4018360220","heartbeat_back_addr":"172.21.15.12:6823/4018360220","heartbeat_front_addr":"172.21.15.12:6821/4018360220","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.546021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.605275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T22:20:45.577859+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":386,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T23:02:48.893 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-12-02T23:02:48.893 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-02T23:02:49.034 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-12-02T23:02:49.034 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-02T23:02:49.034 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-12-02T23:02:49.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-12-02T23:02:49.340 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 0 to deep-scrub 2023-12-02T23:02:49.353 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-12-02T23:02:49.353 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-02T23:02:49.484 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-12-02T23:02:49.485 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-02T23:02:49.485 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-12-02T23:02:49.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-12-02T23:02:49.789 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 1 to deep-scrub 2023-12-02T23:02:49.801 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-12-02T23:02:49.801 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-02T23:02:49.932 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-12-02T23:02:49.932 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-02T23:02:49.932 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-12-02T23:02:49.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-12-02T23:02:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 2 to deep-scrub 2023-12-02T23:02:50.249 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T23:02:50.539 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T23:02:50.539 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T23:02:50.551 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1719,"stamp":"2023-12-02T23:02:48.592517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":448532,"kb_used_data":12272,"kb_used_omap":4,"kb_used_meta":436219,"kb_avail":282666988,"statfs":{"total":289910292480,"available":289450995712,"internally_reserved":0,"allocated":12566528,"data_stored":7294433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":446688591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002324"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:59.805595+0000","last_change":"2023-12-02T22:20:56.564305+0000","last_active":"2023-12-02T22:59:59.805595+0000","last_peered":"2023-12-02T22:59:59.805595+0000","last_clean":"2023-12-02T22:59:59.805595+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T22:59:59.805595+0000","last_undegraded":"2023-12-02T22:59:59.805595+0000","last_fullsized":"2023-12-02T22:59:59.805595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:31:54.028189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":477,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:54.805010+0000","last_change":"2023-12-02T22:20:56.565011+0000","last_active":"2023-12-02T22:59:54.805010+0000","last_peered":"2023-12-02T22:59:54.805010+0000","last_clean":"2023-12-02T22:59:54.805010+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T22:59:54.805010+0000","last_undegraded":"2023-12-02T22:59:54.805010+0000","last_fullsized":"2023-12-02T22:59:54.805010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:00:19.409954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:39.809627+0000","last_change":"2023-12-02T22:20:56.563555+0000","last_active":"2023-12-02T23:00:39.809627+0000","last_peered":"2023-12-02T23:00:39.809627+0000","last_clean":"2023-12-02T23:00:39.809627+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T23:00:39.809627+0000","last_undegraded":"2023-12-02T23:00:39.809627+0000","last_fullsized":"2023-12-02T23:00:39.809627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T09:35:24.530609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809158+0000","last_change":"2023-12-02T22:20:56.562760+0000","last_active":"2023-12-02T23:00:34.809158+0000","last_peered":"2023-12-02T23:00:34.809158+0000","last_clean":"2023-12-02T23:00:34.809158+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T23:00:34.809158+0000","last_undegraded":"2023-12-02T23:00:34.809158+0000","last_fullsized":"2023-12-02T23:00:34.809158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T07:42:42.364951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.815720+0000","last_change":"2023-12-02T22:20:56.562058+0000","last_active":"2023-12-02T23:00:34.815720+0000","last_peered":"2023-12-02T23:00:34.815720+0000","last_clean":"2023-12-02T23:00:34.815720+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T23:00:34.815720+0000","last_undegraded":"2023-12-02T23:00:34.815720+0000","last_fullsized":"2023-12-02T23:00:34.815720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:09:32.035065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:29.788062+0000","last_change":"2023-12-02T22:20:56.563338+0000","last_active":"2023-12-02T23:00:29.788062+0000","last_peered":"2023-12-02T23:00:29.788062+0000","last_clean":"2023-12-02T23:00:29.788062+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T23:00:29.788062+0000","last_undegraded":"2023-12-02T23:00:29.788062+0000","last_fullsized":"2023-12-02T23:00:29.788062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T23:03:15.596248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.807941+0000","last_change":"2023-12-02T22:20:56.565388+0000","last_active":"2023-12-02T22:59:13.807941+0000","last_peered":"2023-12-02T22:59:13.807941+0000","last_clean":"2023-12-02T22:59:13.807941+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T22:59:13.807941+0000","last_undegraded":"2023-12-02T22:59:13.807941+0000","last_fullsized":"2023-12-02T22:59:13.807941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:24:24.875735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:00:34.809141+0000","last_change":"2023-12-02T22:20:56.563039+0000","last_active":"2023-12-02T23:00:34.809141+0000","last_peered":"2023-12-02T23:00:34.809141+0000","last_clean":"2023-12-02T23:00:34.809141+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T23:00:34.809141+0000","last_undegraded":"2023-12-02T23:00:34.809141+0000","last_fullsized":"2023-12-02T23:00:34.809141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:55.542834+0000","last_clean_scrub_stamp":"2023-12-02T22:20:55.542834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T03:37:53.638783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":608,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T22:59:13.101290+0000","last_change":"2023-12-02T22:20:51.698937+0000","last_active":"2023-12-02T22:59:13.101290+0000","last_peered":"2023-12-02T22:59:13.101290+0000","last_clean":"2023-12-02T22:59:13.101290+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T22:59:13.101290+0000","last_undegraded":"2023-12-02T22:59:13.101290+0000","last_fullsized":"2023-12-02T22:59:13.101290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T22:20:50.530145+0000","last_clean_scrub_stamp":"2023-12-02T22:20:50.530145+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T00:30:49.984827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138948,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":94232892,"statfs":{"total":96636764160,"available":96494481408,"internally_reserved":0,"allocated":3792896,"data_stored":2037899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32100000000000001,"15min":0.30599999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":0.441,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.308},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.216,"5min":0.156,"15min":0.156},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.502},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.23899999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41799999999999998,"5min":0.54700000000000004,"15min":0.83699999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.95699999999999996},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129800,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242040,"statfs":{"total":96636764160,"available":96503848960,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32600000000000001,"15min":0.32800000000000001},"min":{"1min":0.224,"5min":0.20699999999999999,"15min":0.16600000000000001},"max":{"1min":0.42799999999999999,"5min":0.46899999999999997,"15min":2.9550000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.48799999999999999,"15min":0.54900000000000004},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34100000000000003,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19400000000000001},"max":{"1min":0.45400000000000001,"5min":0.51600000000000001,"15min":1.129},"last":0.433},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.19500000000000001},"max":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":2.4169999999999998},"last":0.46700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179784,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192056,"statfs":{"total":96636764160,"available":96452665344,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33100000000000002,"15min":0.317},"min":{"1min":0.219,"5min":0.219,"15min":0.16},"max":{"1min":0.504,"5min":0.51000000000000001,"15min":1.5549999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.34300000000000003},"min":{"1min":0.217,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":0.504,"5min":0.53700000000000003,"15min":0.67800000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":2.0070000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.37},"min":{"1min":0.32900000000000001,"5min":0.26100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:02:50.554 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.554 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.554 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.554 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.554 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.555 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.555 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.555 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-02T22:20:55.542834+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=55, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.555 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-02T22:20:50.530145+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=22, tm_min=20, tm_sec=50, tm_wday=5, tm_yday=336, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=2, tm_hour=23, tm_min=2, tm_sec=47, tm_wday=5, tm_yday=336, tm_isdst=0) 2023-12-02T23:02:50.555 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-02T23:03:10.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T23:03:10.859 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-02T23:03:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-02T23:03:10.871 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1730,"stamp":"2023-12-02T23:03:10.597124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":452680,"kb_used_data":12272,"kb_used_omap":4,"kb_used_meta":440379,"kb_avail":282662840,"statfs":{"total":289910292480,"available":289446748160,"internally_reserved":0,"allocated":12566528,"data_stored":7294433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":450948431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002585"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":486,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:54.765714+0000","last_change":"2023-12-02T23:02:54.765737+0000","last_active":"2023-12-02T23:02:54.765714+0000","last_peered":"2023-12-02T23:02:54.765714+0000","last_clean":"2023-12-02T23:02:54.765714+0000","last_became_active":"2023-12-02T22:20:56.563318+0000","last_became_peered":"2023-12-02T22:20:56.563318+0000","last_unstale":"2023-12-02T23:02:54.765714+0000","last_undegraded":"2023-12-02T23:02:54.765714+0000","last_fullsized":"2023-12-02T23:02:54.765714+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:54.765594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:54.765594+0000","last_clean_scrub_stamp":"2023-12-02T23:02:54.765594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:45:52.368160+0000","scrub_duration":0.021061136000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":486,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:53.703514+0000","last_change":"2023-12-02T23:02:53.703537+0000","last_active":"2023-12-02T23:02:53.703514+0000","last_peered":"2023-12-02T23:02:53.703514+0000","last_clean":"2023-12-02T23:02:53.703514+0000","last_became_active":"2023-12-02T22:20:56.564588+0000","last_became_peered":"2023-12-02T22:20:56.564588+0000","last_unstale":"2023-12-02T23:02:53.703514+0000","last_undegraded":"2023-12-02T23:02:53.703514+0000","last_fullsized":"2023-12-02T23:02:53.703514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:53.703402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:53.703402+0000","last_clean_scrub_stamp":"2023-12-02T23:02:53.703402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:14:17.703564+0000","scrub_duration":0.0081980750000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":487,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:58.778614+0000","last_change":"2023-12-02T23:02:58.778638+0000","last_active":"2023-12-02T23:02:58.778614+0000","last_peered":"2023-12-02T23:02:58.778614+0000","last_clean":"2023-12-02T23:02:58.778614+0000","last_became_active":"2023-12-02T22:20:56.563149+0000","last_became_peered":"2023-12-02T22:20:56.563149+0000","last_unstale":"2023-12-02T23:02:58.778614+0000","last_undegraded":"2023-12-02T23:02:58.778614+0000","last_fullsized":"2023-12-02T23:02:58.778614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:58.778493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:58.778493+0000","last_clean_scrub_stamp":"2023-12-02T23:02:58.778493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T02:20:44.735501+0000","scrub_duration":0.018327062000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":487,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:59.764271+0000","last_change":"2023-12-02T23:02:59.764290+0000","last_active":"2023-12-02T23:02:59.764271+0000","last_peered":"2023-12-02T23:02:59.764271+0000","last_clean":"2023-12-02T23:02:59.764271+0000","last_became_active":"2023-12-02T22:20:56.562431+0000","last_became_peered":"2023-12-02T22:20:56.562431+0000","last_unstale":"2023-12-02T23:02:59.764271+0000","last_undegraded":"2023-12-02T23:02:59.764271+0000","last_fullsized":"2023-12-02T23:02:59.764271+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:59.764181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:59.764181+0000","last_clean_scrub_stamp":"2023-12-02T23:02:59.764181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T05:16:43.867607+0000","scrub_duration":0.00146806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":487,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:49.391658+0000","last_change":"2023-12-02T23:02:49.391688+0000","last_active":"2023-12-02T23:02:49.391658+0000","last_peered":"2023-12-02T23:02:49.391658+0000","last_clean":"2023-12-02T23:02:49.391658+0000","last_became_active":"2023-12-02T22:20:56.561708+0000","last_became_peered":"2023-12-02T22:20:56.561708+0000","last_unstale":"2023-12-02T23:02:49.391658+0000","last_undegraded":"2023-12-02T23:02:49.391658+0000","last_fullsized":"2023-12-02T23:02:49.391658+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:49.391565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:49.391565+0000","last_clean_scrub_stamp":"2023-12-02T23:02:49.391565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T23:59:50.110671+0000","scrub_duration":0.0053000690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":487,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:52.340654+0000","last_change":"2023-12-02T23:02:52.340680+0000","last_active":"2023-12-02T23:02:52.340654+0000","last_peered":"2023-12-02T23:02:52.340654+0000","last_clean":"2023-12-02T23:02:52.340654+0000","last_became_active":"2023-12-02T22:20:56.563052+0000","last_became_peered":"2023-12-02T22:20:56.563052+0000","last_unstale":"2023-12-02T23:02:52.340654+0000","last_undegraded":"2023-12-02T23:02:52.340654+0000","last_fullsized":"2023-12-02T23:02:52.340654+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:52.340539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:52.340539+0000","last_clean_scrub_stamp":"2023-12-02T23:02:52.340539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T04:34:39.722049+0000","scrub_duration":0.019273617999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":488,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:54.326541+0000","last_change":"2023-12-02T23:02:54.326541+0000","last_active":"2023-12-02T23:02:54.326541+0000","last_peered":"2023-12-02T23:02:54.326541+0000","last_clean":"2023-12-02T23:02:54.326541+0000","last_became_active":"2023-12-02T22:20:56.564462+0000","last_became_peered":"2023-12-02T22:20:56.564462+0000","last_unstale":"2023-12-02T23:02:54.326541+0000","last_undegraded":"2023-12-02T23:02:54.326541+0000","last_fullsized":"2023-12-02T23:02:54.326541+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:54.326396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:54.326396+0000","last_clean_scrub_stamp":"2023-12-02T23:02:54.326396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T02:45:00.047706+0000","scrub_duration":0.011185495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":487,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:55.741219+0000","last_change":"2023-12-02T23:02:55.741243+0000","last_active":"2023-12-02T23:02:55.741219+0000","last_peered":"2023-12-02T23:02:55.741219+0000","last_clean":"2023-12-02T23:02:55.741219+0000","last_became_active":"2023-12-02T22:20:56.562734+0000","last_became_peered":"2023-12-02T22:20:56.562734+0000","last_unstale":"2023-12-02T23:02:55.741219+0000","last_undegraded":"2023-12-02T23:02:55.741219+0000","last_fullsized":"2023-12-02T23:02:55.741219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T23:02:55.741102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T23:02:55.741102+0000","last_clean_scrub_stamp":"2023-12-02T23:02:55.741102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T01:13:44.812393+0000","scrub_duration":0.020151144999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"461'99","reported_seq":620,"reported_epoch":461,"state":"active+clean","last_fresh":"2023-12-02T23:02:52.719400+0000","last_change":"2023-12-02T23:02:52.719400+0000","last_active":"2023-12-02T23:02:52.719400+0000","last_peered":"2023-12-02T23:02:52.719400+0000","last_clean":"2023-12-02T23:02:52.719400+0000","last_became_active":"2023-12-02T22:20:51.697965+0000","last_became_peered":"2023-12-02T22:20:51.697965+0000","last_unstale":"2023-12-02T23:02:52.719400+0000","last_undegraded":"2023-12-02T23:02:52.719400+0000","last_fullsized":"2023-12-02T23:02:52.719400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'99","last_scrub_stamp":"2023-12-02T23:02:52.719276+0000","last_deep_scrub":"461'99","last_deep_scrub_stamp":"2023-12-02T23:02:52.719276+0000","last_clean_scrub_stamp":"2023-12-02T23:02:52.719276+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T08:17:44.747404+0000","scrub_duration":0.0055190830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138944,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":94232896,"statfs":{"total":96636764160,"available":96494485504,"internally_reserved":0,"allocated":3792896,"data_stored":2037899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32100000000000001,"15min":0.30599999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":0.441,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.216,"5min":0.156,"15min":0.156},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.502},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Dec 2 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32400000000000001,"15min":0.31900000000000001},"min":{"1min":0.23899999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41799999999999998,"5min":0.54700000000000004,"15min":0.83699999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.95699999999999996},"last":0.433}]}]},{"osd":1,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129796,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94242044,"statfs":{"total":96636764160,"available":96503853056,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32600000000000001,"15min":0.32800000000000001},"min":{"1min":0.224,"5min":0.20699999999999999,"15min":0.16600000000000001},"max":{"1min":0.42799999999999999,"5min":0.46899999999999997,"15min":2.9550000000000001},"last":0.378},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.48799999999999999,"15min":0.54900000000000004},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Dec 2 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34100000000000003,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19400000000000001},"max":{"1min":0.45400000000000001,"5min":0.51600000000000001,"15min":1.129},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.19500000000000001},"max":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":2.4169999999999998},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":183940,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":94187900,"statfs":{"total":96636764160,"available":96448409600,"internally_reserved":0,"allocated":4386816,"data_stored":2628267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":183957957},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 2 23:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32800000000000001,"15min":0.316},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":0.47199999999999998,"5min":0.51000000000000001,"15min":1.5549999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35799999999999998,"15min":0.34200000000000003},"min":{"1min":0.22600000000000001,"5min":0.217,"15min":0.186},"max":{"1min":0.49199999999999999,"5min":0.53700000000000003,"15min":0.67800000000000005},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Dec 2 23:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34899999999999998,"15min":0.33400000000000002},"min":{"1min":0.248,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.55200000000000005,"5min":0.57699999999999996,"15min":2.0070000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.21299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T23:03:10.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-12-02T23:03:11.184 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T23:03:11.185 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-02T23:03:11.185 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-02T23:03:11.186 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-02T23:03:11.186 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:03:11.240 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-02T23:03:11.241 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-02T23:03:11.241 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:03:11.288 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-02T23:03:11.289 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-02T23:03:11.289 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:03:11.334 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-02T23:03:11.334 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-02T23:03:11.334 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-02T23:03:11.334 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:03:11.359 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-02T23:03:11.359 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-02T23:03:11.360 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-02T23:03:11.360 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T23:03:11.413 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-02T23:03:11.413 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-02T23:03:11.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-02T23:03:11.486 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T23:03:11.486 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-02T23:03:11.701 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T23:03:11.701 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-02T23:03:11.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T23:03:11.777 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-02T23:03:11.849 INFO:tasks.ceph:Archiving mon data... 2023-12-02T23:03:11.850 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/data/cluster1.mon.a.tgz 2023-12-02T23:03:11.850 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-12-02T23:03:11.864 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.vdxG3hL5KI 2023-12-02T23:03:11.865 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.vdxG3hL5KI 2023-12-02T23:03:12.004 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.vdxG3hL5KI 2023-12-02T23:03:12.093 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.vdxG3hL5KI is 292KB 2023-12-02T23:03:12.188 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.vdxG3hL5KI 2023-12-02T23:03:12.203 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-02T23:03:12.203 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T23:03:12.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T23:03:12.329 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-02T23:03:12.462 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-02T23:03:12.462 INFO:tasks.ceph:Archiving crash dumps... 2023-12-02T23:03:12.464 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/remote/smithi012/crash 2023-12-02T23:03:12.465 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-02T23:03:12.496 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/remote/smithi170/crash 2023-12-02T23:03:12.497 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-02T23:03:12.571 INFO:tasks.ceph:Compressing logs... 2023-12-02T23:03:12.571 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-02T23:03:12.573 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-02T23:03:12.601 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89956.log 2023-12-02T23:03:12.602 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89956.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89956.log.gz 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89746.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92288.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89746.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80221.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89746.log.gz 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89869.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92288.log.gz 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.tmp-client.admin.80221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89620.log 2023-12-02T23:03:12.606 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80221.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89869.log.gzgzip 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.90250.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92204.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89620.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.90250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90250.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89788.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92204.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85127.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89788.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster1-client.admin.89788.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.89995.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.85127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92498.log 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85127.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89995.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.89995.log.gz 2023-12-02T23:03:12.607 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/cluster1-client.admin.89480.log 2023-12-02T23:03:12.621 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92498.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89480.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90208.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85057.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/cluster1-client.admin.90208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90121.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90208.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.85057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89704.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85057.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.90121.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90121.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84920.log 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89704.log.gz 2023-12-02T23:03:12.622 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92024.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.84920.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84920.log.gz 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92456.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92024.log.gz 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84722.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92456.log.gz 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92246.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.84722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84722.log.gz 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84880.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92246.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92246.log.gz 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.90082.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.84880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84880.log.gz 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89564.log 2023-12-02T23:03:12.623 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.90082.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90082.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89564.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84960.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90292.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.84960.log: 88.1% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84960.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89868.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5/var/log/ceph/cluster1-client.admin.90292.log: --verbose -- /var/log/ceph/cluster1-client.admin.90122.log 0.0% 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.90292.log.gz/var/log/ceph/cluster1-client.admin.89868.log: 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89868.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89662.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.90122.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90122.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.89870.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89662.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89662.log.gz 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.92159.log 2023-12-02T23:03:12.624 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.89870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89870.log.gz 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92330.log 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/cluster1-client.admin.92159.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92159.log.gz 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/cluster1-client.admin.89996.log 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92330.log.gz 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92414.log 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/cluster1-client.admin.89996.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89996.log.gz 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/cluster1-client.admin.92538.log 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92414.log.gz 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-12-02T23:03:12.625 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/cluster1-client.admin.92538.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92538.log.gz 2023-12-02T23:03:12.626 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/cluster1-client.admin.92372.log 2023-12-02T23:03:12.626 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92067.log 2023-12-02T23:03:12.626 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92372.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92111.log 2023-12-02T23:03:12.626 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.92372.log.gz 2023-12-02T23:03:12.626 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92067.log.gz 2023-12-02T23:03:12.626 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cluster1-client.admin.92111.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92111.log.gz 2023-12-02T23:03:12.660 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121371.log 2023-12-02T23:03:12.660 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109864.log 2023-12-02T23:03:12.660 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110825.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121371.log.gz 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94510.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109864.log.gz 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102475.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99950.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102475.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.94510.log.gz 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113442.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102475.log.gz 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102312.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106534.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100422.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100844.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115484.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.100844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114938.log 2023-12-02T23:03:12.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96787.log 2023-12-02T23:03:12.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90475.log 2023-12-02T23:03:12.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101503.log 2023-12-02T23:03:12.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118240.log 2023-12-02T23:03:12.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108919.log 2023-12-02T23:03:12.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112027.log 2023-12-02T23:03:12.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97525.log 2023-12-02T23:03:12.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96061.log 2023-12-02T23:03:12.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102983.log 2023-12-02T23:03:12.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104683.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128551.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121101.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112027.log.gz 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91289.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128551.log.gz 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106100.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121101.log.gz 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120426.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116467.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124770.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120426.log.gz 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122490.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119906.log 2023-12-02T23:03:12.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122490.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124770.log.gz 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127177.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122490.log.gz 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121882.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109381.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127177.log.gz 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119906.log.gz 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115597.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121882.log.gz 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116805.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.121722.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121304.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108919.log.gz 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.121722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95018.log 2023-12-02T23:03:12.699 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107655.log 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121304.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109381.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95018.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121304.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111672.log 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95018.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107655.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116805.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111672.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114938.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99238.log 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113442.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115484.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111672.log.gz 2023-12-02T23:03:12.700 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120133.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116580.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99238.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110825.log.gz 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.120133.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115597.log.gz 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118488.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93767.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123712.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116467.log.gz 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115918.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118488.log.gz 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.108114.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121702.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118240.log.gz 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107366.log 2023-12-02T23:03:12.701 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125011.log 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.108114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113745.log 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102983.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115918.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.96787.log.gz 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91289.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121702.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.100844.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99238.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109160.log 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106534.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96061.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125011.log: /var/log/ceph/cluster1-client.mirror.113745.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121702.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113745.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125011.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101503.log.gz 2023-12-02T23:03:12.702 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6%gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107655.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.100422.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.104182.log 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109160.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116580.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.121722.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.107366.log: -- /var/log/ceph/cluster2-client.mirror.98287.log 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107366.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104182.log: 87.7% 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120133.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106100.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.104683.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.104182.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124749.log 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.98287.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109160.log.gz 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.119183.log 2023-12-02T23:03:12.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124749.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124749.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster1-client.mirror.97998.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98287.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118741.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119183.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97998.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123653.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101610.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118741.log: /var/log/ceph/cluster2-client.mirror.123653.log: 80.5% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118741.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.123653.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121191.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90259.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101610.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121191.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121191.log.gz 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109517.log 2023-12-02T23:03:12.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/cluster2-client.admin.90259.log: -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90259.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.112193.log 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109517.log: gzip -5 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109517.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.125073.log 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93413.log 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125073.log: gzip 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125073.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110620.log 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112193.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110732.log 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110620.log: gzip -5 --verbose -- 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97998.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.108573.log 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93413.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97525.log.gz 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110732.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121619.log 2023-12-02T23:03:12.705 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110732.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125793.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108573.log: /var/log/ceph/cluster1-client.mirror.121619.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110620.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121972.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121619.log.gz 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108573.log.gz 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126571.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125793.log.gz 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125169.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121972.log.gz 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.90747.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106555.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.3.90747.log: -- replaced with /var/log/ceph/cluster1-client.mirror.125169.log.gz 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103434.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117496.log 2023-12-02T23:03:12.706 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93249.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126571.log.gz/var/log/ceph/cluster2-client.mirror.117496.log: 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126651.log 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106555.log.gz 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103434.log: /var/log/ceph/cluster2-client.mirror.93249.log: 89.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116116.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117496.log.gz 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103434.log.gz 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126486.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99928.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126486.log: gzip -5 --verbose 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126486.log.gz 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.116055.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108203.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.116055.log: -- replaced with /var/log/ceph/cluster2-client.mirror.99928.log.gz 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111491.log 2023-12-02T23:03:12.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108203.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116055.log.gz 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108203.log.gz 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119534.log 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113326.log 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119534.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119534.log.gz 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95087.log 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113326.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111491.log.gz 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104405.log 89.7% 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.113326.log.gz 2023-12-02T23:03:12.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119286.log 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104405.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116116.log.gz 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104405.log.gz 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109772.log 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114852.log 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109772.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119286.log.gz 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127132.log 2023-12-02T23:03:12.718 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.95087.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109772.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90091.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.127132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99800.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114852.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112309.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99800.log: /var/log/ceph/cluster1-client.admin.90091.log: 80.4% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90091.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99800.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127132.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99859.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115665.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99859.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112309.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95976.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118343.log 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115665.log: /var/log/ceph/cluster2-client.mirror.95976.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115665.log.gz 2023-12-02T23:03:12.719 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104336.log 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118343.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93249.log.gz 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98900.log 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104336.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104336.log.gz 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105313.log 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98900.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98900.log.gz 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95976.log.gz 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.104485.log 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.105313.log: -- /var/log/ceph/cluster2-client.mirror.96019.log 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105313.log.gz/var/log/ceph/cluster2-client.mirror.2.104485.log: 2023-12-02T23:03:12.720 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109023.log 2023-12-02T23:03:12.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119741.log 2023-12-02T23:03:12.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109023.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96019.log.gz 2023-12-02T23:03:12.723 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109023.log.gz 2023-12-02T23:03:12.723 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126551.log 2023-12-02T23:03:12.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119741.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118343.log.gz 2023-12-02T23:03:12.724 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119741.log.gz 2023-12-02T23:03:12.724 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107700.log 2023-12-02T23:03:12.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126551.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126551.log.gz 2023-12-02T23:03:12.725 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107193.log 2023-12-02T23:03:12.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107700.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107700.log.gz 2023-12-02T23:03:12.727 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108049.log 2023-12-02T23:03:12.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107193.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107193.log.gz 2023-12-02T23:03:12.729 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104066.log 2023-12-02T23:03:12.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108049.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108049.log.gz 2023-12-02T23:03:12.730 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117145.log 2023-12-02T23:03:12.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104066.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.108114.log.gz 2023-12-02T23:03:12.743 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96655.log 2023-12-02T23:03:12.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117145.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.90747.log.gz 2023-12-02T23:03:12.744 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117145.log.gz 2023-12-02T23:03:12.745 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125766.log 2023-12-02T23:03:12.745 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96655.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96655.log.gz 2023-12-02T23:03:12.746 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106715.log 2023-12-02T23:03:12.747 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125766.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125766.log.gz 2023-12-02T23:03:12.747 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123802.log 2023-12-02T23:03:12.749 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106715.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106715.log.gz 2023-12-02T23:03:12.749 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113487.log 2023-12-02T23:03:12.750 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123802.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123802.log.gz 2023-12-02T23:03:12.750 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125192.log 2023-12-02T23:03:12.753 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113487.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113487.log.gz 2023-12-02T23:03:12.753 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109340.log 2023-12-02T23:03:12.753 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125192.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125192.log.gz 2023-12-02T23:03:12.757 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116740.log 2023-12-02T23:03:12.757 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109340.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.104485.log.gz 2023-12-02T23:03:12.757 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109340.log.gz 2023-12-02T23:03:12.757 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.103573.log 2023-12-02T23:03:12.757 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117579.log 2023-12-02T23:03:12.757 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116740.log.gz 2023-12-02T23:03:12.758 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.3.103573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126530.log 2023-12-02T23:03:12.758 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117579.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117579.log.gz 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110483.log 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126530.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126530.log.gz 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104161.log 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110483.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110483.log.gz 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114894.log 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104161.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104161.log.gz 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111142.log 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114894.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114894.log.gz 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95130.log 2023-12-02T23:03:12.766 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111142.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111142.log.gz 2023-12-02T23:03:12.767 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125981.log 2023-12-02T23:03:12.769 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95130.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95130.log.gz 2023-12-02T23:03:12.769 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105667.log 2023-12-02T23:03:12.770 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125981.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125981.log.gz 2023-12-02T23:03:12.770 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128509.log 2023-12-02T23:03:12.773 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105667.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105667.log.gz 2023-12-02T23:03:12.774 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109244.log 2023-12-02T23:03:12.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128509.log.gz 2023-12-02T23:03:12.774 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122355.log 2023-12-02T23:03:12.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109244.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109244.log.gz 2023-12-02T23:03:12.775 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98147.log 2023-12-02T23:03:12.775 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122355.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122355.log.gz 2023-12-02T23:03:12.775 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103377.log 2023-12-02T23:03:12.777 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98147.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98147.log.gz 2023-12-02T23:03:12.777 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120359.log 2023-12-02T23:03:12.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95931.log 2023-12-02T23:03:12.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120359.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120359.log.gz 2023-12-02T23:03:12.804 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97978.log 2023-12-02T23:03:12.804 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95931.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95931.log.gz 2023-12-02T23:03:12.816 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125213.log 2023-12-02T23:03:12.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97978.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93905.log 2023-12-02T23:03:12.832 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.97978.log.gz 2023-12-02T23:03:12.844 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125213.log: 91.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128131.log 2023-12-02T23:03:12.844 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.125213.log.gz 2023-12-02T23:03:12.851 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112571.log 2023-12-02T23:03:12.852 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128131.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93905.log.gz 2023-12-02T23:03:12.864 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128131.log.gz 2023-12-02T23:03:12.865 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126859.log 2023-12-02T23:03:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112571.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112571.log.gz 2023-12-02T23:03:12.874 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89377.log 2023-12-02T23:03:12.884 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111556.log 2023-12-02T23:03:12.884 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89377.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89377.log.gz 2023-12-02T23:03:12.887 INFO:teuthology.orchestra.run.smithi170.stderr: 75.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126859.log.gz 2023-12-02T23:03:12.900 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106221.log 2023-12-02T23:03:12.903 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111556.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111556.log.gz 2023-12-02T23:03:12.910 INFO:teuthology.orchestra.run.smithi170.stderr: 92.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90049.log 2023-12-02T23:03:12.910 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106221.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106221.log.gz 2023-12-02T23:03:12.910 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.103573.log.gz 2023-12-02T23:03:12.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119017.log 2023-12-02T23:03:12.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.90049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90049.log.gz 2023-12-02T23:03:12.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91357.log 2023-12-02T23:03:12.914 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119017.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119017.log.gz 2023-12-02T23:03:12.915 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97129.log 2023-12-02T23:03:12.915 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127963.log 2023-12-02T23:03:12.916 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97129.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97129.log.gz 2023-12-02T23:03:12.918 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93847.log 2023-12-02T23:03:12.918 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127963.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127963.log.gz 2023-12-02T23:03:12.918 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129812.log 2023-12-02T23:03:12.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103845.log 2023-12-02T23:03:12.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129812.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129812.log.gz 2023-12-02T23:03:12.920 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.125501.log 2023-12-02T23:03:12.921 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103845.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103845.log.gz 2023-12-02T23:03:12.922 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108227.log 2023-12-02T23:03:12.924 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.3.125501.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.125501.log.gz 2023-12-02T23:03:12.925 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117227.log 2023-12-02T23:03:12.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108227.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108227.log.gz 2023-12-02T23:03:12.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97396.log 2023-12-02T23:03:12.926 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117227.log: 56.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91357.log.gz 2023-12-02T23:03:12.928 INFO:teuthology.orchestra.run.smithi170.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117227.log.gz 2023-12-02T23:03:12.929 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115300.log 2023-12-02T23:03:12.935 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97396.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97396.log.gz 2023-12-02T23:03:12.935 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94356.log 2023-12-02T23:03:12.938 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115300.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115300.log.gz 2023-12-02T23:03:12.938 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124848.log 2023-12-02T23:03:12.939 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-02T23:03:12.940 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94356.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94356.log.gz 2023-12-02T23:03:12.940 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103477.log 2023-12-02T23:03:12.941 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93847.log.gz 2023-12-02T23:03:12.942 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124848.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124848.log.gz 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127501.log 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103477.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103477.log.gz 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99758.log 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127501.log.gz 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92600.log 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99758.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99758.log.gz 2023-12-02T23:03:12.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120291.log 2023-12-02T23:03:12.949 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91413.log 2023-12-02T23:03:12.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120291.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120291.log.gz 2023-12-02T23:03:12.950 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130862.log 2023-12-02T23:03:12.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92161.log 2023-12-02T23:03:12.951 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130862.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130862.log.gz 2023-12-02T23:03:12.951 INFO:teuthology.orchestra.run.smithi170.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91413.log.gz 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89710.log 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111962.log 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89710.log.gz 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115713.log 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111962.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111962.log.gz 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126922.log 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115713.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115713.log.gz 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129097.log 2023-12-02T23:03:12.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126922.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126922.log.gz 2023-12-02T23:03:12.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118893.log 2023-12-02T23:03:12.961 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129097.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129097.log.gz 2023-12-02T23:03:12.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114631.log 2023-12-02T23:03:12.962 INFO:teuthology.orchestra.run.smithi012.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-02T23:03:12.964 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118893.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118893.log.gz 2023-12-02T23:03:12.965 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.116909.log 2023-12-02T23:03:12.966 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114631.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114631.log.gz 2023-12-02T23:03:12.966 INFO:teuthology.orchestra.run.smithi170.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92161.log.gz 2023-12-02T23:03:12.967 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92600.log.gz 2023-12-02T23:03:12.968 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120381.log 2023-12-02T23:03:12.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.116909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117248.log 2023-12-02T23:03:12.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120381.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120381.log.gz 2023-12-02T23:03:12.978 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.116909.log.gz 2023-12-02T23:03:12.979 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115689.log 2023-12-02T23:03:12.980 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117248.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117248.log.gz 2023-12-02T23:03:12.980 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91374.log 2023-12-02T23:03:12.981 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115689.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115689.log.gz 2023-12-02T23:03:12.981 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92401.log 2023-12-02T23:03:12.982 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125132.log 2023-12-02T23:03:12.982 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105868.log 2023-12-02T23:03:12.984 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125132.log: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91374.log.gz 2023-12-02T23:03:12.993 INFO:teuthology.orchestra.run.smithi170.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92401.log.gz 2023-12-02T23:03:12.994 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107414.log 2023-12-02T23:03:13.004 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105868.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105868.log.gz 2023-12-02T23:03:13.005 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89583.log 2023-12-02T23:03:13.005 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107414.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107414.log.gz 2023-12-02T23:03:13.019 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.93502.log 2023-12-02T23:03:13.019 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89583.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89583.log.gz 2023-12-02T23:03:13.020 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114030.log 2023-12-02T23:03:13.037 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.1.93502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129980.log 2023-12-02T23:03:13.038 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114030.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114030.log.gz 2023-12-02T23:03:13.045 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107991.log 2023-12-02T23:03:13.045 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129980.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129980.log.gz 2023-12-02T23:03:13.050 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102312.log.gz 2023-12-02T23:03:13.050 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120269.log 2023-12-02T23:03:13.052 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107991.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107991.log.gz 2023-12-02T23:03:13.052 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96531.log 2023-12-02T23:03:13.052 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120269.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120269.log.gz 2023-12-02T23:03:13.052 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120180.log 2023-12-02T23:03:13.053 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96531.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.96531.log.gz 2023-12-02T23:03:13.053 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123612.log 2023-12-02T23:03:13.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120180.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120180.log.gz 2023-12-02T23:03:13.054 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119203.log 2023-12-02T23:03:13.055 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123612.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123612.log.gz 2023-12-02T23:03:13.055 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98227.log 2023-12-02T23:03:13.058 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119203.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119203.log.gz 2023-12-02T23:03:13.058 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121641.log 2023-12-02T23:03:13.059 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98227.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98227.log.gz 2023-12-02T23:03:13.060 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106462.log 2023-12-02T23:03:13.060 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121641.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121641.log.gz 2023-12-02T23:03:13.060 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124188.log 2023-12-02T23:03:13.061 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106462.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106462.log.gz 2023-12-02T23:03:13.061 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130316.log 2023-12-02T23:03:13.062 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124188.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124188.log.gz 2023-12-02T23:03:13.063 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94994.log 2023-12-02T23:03:13.063 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130316.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130316.log.gz 2023-12-02T23:03:13.063 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100172.log 2023-12-02T23:03:13.064 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94994.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94994.log.gz 2023-12-02T23:03:13.064 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106826.log 2023-12-02T23:03:13.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100172.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100172.log.gz 2023-12-02T23:03:13.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122017.log 2023-12-02T23:03:13.067 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106826.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106826.log.gz 2023-12-02T23:03:13.067 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110306.log 2023-12-02T23:03:13.067 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122017.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122017.log.gz 2023-12-02T23:03:13.068 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112593.log 2023-12-02T23:03:13.069 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110306.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110306.log.gz 2023-12-02T23:03:13.069 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116692.log 2023-12-02T23:03:13.070 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112593.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112593.log.gz 2023-12-02T23:03:13.071 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106916.log 2023-12-02T23:03:13.071 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116692.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116692.log.gz 2023-12-02T23:03:13.072 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129392.log 2023-12-02T23:03:13.072 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106916.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.106916.log.gz 2023-12-02T23:03:13.072 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129181.log 2023-12-02T23:03:13.073 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129392.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129392.log.gz 2023-12-02T23:03:13.074 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96763.log 2023-12-02T23:03:13.074 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129181.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129181.log.gz 2023-12-02T23:03:13.074 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101875.log 2023-12-02T23:03:13.075 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96763.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96763.log.gz 2023-12-02T23:03:13.075 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99908.log 2023-12-02T23:03:13.076 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101875.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101875.log.gz 2023-12-02T23:03:13.076 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94025.log 2023-12-02T23:03:13.077 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99908.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99908.log.gz 2023-12-02T23:03:13.078 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102702.log 2023-12-02T23:03:13.079 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8%/var/log/ceph/cluster1-client.mirror.94025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120989.log 2023-12-02T23:03:13.081 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102702.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94025.log.gz 2023-12-02T23:03:13.081 INFO:teuthology.orchestra.run.smithi170.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102702.log.gz 2023-12-02T23:03:13.081 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99950.log.gz 2023-12-02T23:03:13.081 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103828.log 2023-12-02T23:03:13.083 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120989.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120989.log.gz 2023-12-02T23:03:13.083 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105337.log 2023-12-02T23:03:13.083 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103828.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103828.log.gz 2023-12-02T23:03:13.083 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122559.log 2023-12-02T23:03:13.084 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105337.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105337.log.gz 2023-12-02T23:03:13.085 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125399.log 2023-12-02T23:03:13.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122559.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122559.log.gz 2023-12-02T23:03:13.086 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103414.log 2023-12-02T23:03:13.086 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125399.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.93502.log.gz 2023-12-02T23:03:13.103 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103377.log.gz 2023-12-02T23:03:13.104 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104748.log 2023-12-02T23:03:13.106 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103414.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103414.log.gz 2023-12-02T23:03:13.106 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111790.log 2023-12-02T23:03:13.108 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104748.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104748.log.gz 2023-12-02T23:03:13.108 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105626.log 2023-12-02T23:03:13.110 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111790.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111790.log.gz 2023-12-02T23:03:13.110 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90667.log 2023-12-02T23:03:13.112 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105626.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105626.log.gz 2023-12-02T23:03:13.112 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120224.log 2023-12-02T23:03:13.114 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119121.log 2023-12-02T23:03:13.114 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120224.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120224.log.gz 2023-12-02T23:03:13.114 INFO:teuthology.orchestra.run.smithi170.stderr: 85.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90667.log.gz 2023-12-02T23:03:13.114 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97482.log 2023-12-02T23:03:13.115 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119121.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119121.log.gz 2023-12-02T23:03:13.115 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126088.log 2023-12-02T23:03:13.121 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97482.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97482.log.gz 2023-12-02T23:03:13.122 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97568.log 2023-12-02T23:03:13.123 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126088.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126088.log.gz 2023-12-02T23:03:13.123 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104066.log.gz 2023-12-02T23:03:13.123 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117744.log 2023-12-02T23:03:13.130 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97568.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97568.log.gz 2023-12-02T23:03:13.130 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115776.log 2023-12-02T23:03:13.130 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111467.log 2023-12-02T23:03:13.131 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115776.log: /var/log/ceph/cluster2-client.mirror.117744.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117744.log.gz 2023-12-02T23:03:13.132 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115776.log.gz 2023-12-02T23:03:13.132 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110845.log 2023-12-02T23:03:13.132 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128089.log 2023-12-02T23:03:13.132 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111467.log: /var/log/ceph/cluster1-client.mirror.110845.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111467.log.gz 2023-12-02T23:03:13.135 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121836.log 2023-12-02T23:03:13.135 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128089.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128089.log.gz 2023-12-02T23:03:13.135 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127417.log 2023-12-02T23:03:13.135 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121836.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121836.log.gz 2023-12-02T23:03:13.135 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118033.log 2023-12-02T23:03:13.135 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127417.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127417.log.gz 2023-12-02T23:03:13.136 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103573.log 2023-12-02T23:03:13.139 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118033.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118033.log.gz 2023-12-02T23:03:13.139 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129938.log 2023-12-02T23:03:13.151 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103573.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110845.log.gz 2023-12-02T23:03:13.152 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115983.log 2023-12-02T23:03:13.152 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129938.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129938.log.gz 2023-12-02T23:03:13.163 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114410.log 2023-12-02T23:03:13.165 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115983.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115983.log.gz 2023-12-02T23:03:13.170 INFO:teuthology.orchestra.run.smithi170.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90475.log.gz 2023-12-02T23:03:13.171 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96681.log 2023-12-02T23:03:13.173 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114410.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114410.log.gz 2023-12-02T23:03:13.173 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104728.log 2023-12-02T23:03:13.175 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96681.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96681.log.gz 2023-12-02T23:03:13.175 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123867.log 2023-12-02T23:03:13.177 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104728.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104728.log.gz 2023-12-02T23:03:13.177 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96986.log 2023-12-02T23:03:13.178 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123867.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123867.log.gz 2023-12-02T23:03:13.178 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97955.log 2023-12-02T23:03:13.185 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96986.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96986.log.gz 2023-12-02T23:03:13.185 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110041.log 2023-12-02T23:03:13.190 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97955.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97955.log.gz 2023-12-02T23:03:13.191 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111395.log 2023-12-02T23:03:13.192 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110041.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110041.log.gz 2023-12-02T23:03:13.192 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96852.log 2023-12-02T23:03:13.193 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111395.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123712.log.gz 2023-12-02T23:03:13.193 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111395.log.gz 2023-12-02T23:03:13.193 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120944.log 2023-12-02T23:03:13.195 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129434.log 2023-12-02T23:03:13.195 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96852.log: 74.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96852.log.gz 2023-12-02T23:03:13.195 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117124.log 2023-12-02T23:03:13.195 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120944.log.gz 2023-12-02T23:03:13.195 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129434.log.gz 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98107.log 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115235.log 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117124.log: /var/log/ceph/cluster2-client.mirror.98107.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117124.log.gz 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103573.log.gz 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98107.log.gz 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92439.log 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94066.log 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115235.log.gz/var/log/ceph/cluster2-client.mirror.92439.log: 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94394.log 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103808.log 2023-12-02T23:03:13.206 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98528.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125110.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98528.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94394.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121995.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125110.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125110.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117992.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121995.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121995.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119410.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117992.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98528.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117992.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster1-client.mirror.126695.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102517.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126695.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119410.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126695.log.gz 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114323.log 2023-12-02T23:03:13.207 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102517.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102517.log.gz 2023-12-02T23:03:13.208 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96397.log 2023-12-02T23:03:13.208 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114323.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114323.log.gz 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122467.log 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129266.log 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122467.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122467.log.gz 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112776.log 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129266.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129266.log.gz 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100405.log 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119637.log 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92439.log.gz 2023-12-02T23:03:13.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100405.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100405.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92531.log 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4%/var/log/ceph/cluster2-client.mirror.119637.log: -- replaced with /var/log/ceph/cluster2-client.mirror.112776.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117434.log 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111443.log 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117434.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96397.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105554.log 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117434.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116602.log 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111443.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119637.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116602.log: /var/log/ceph/cluster1-client.mirror.105554.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105554.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 89.6% --verbose -- /var/log/ceph/cluster1-client.mirror.92555.log 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116602.log.gz 2023-12-02T23:03:13.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117661.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92531.log.gz 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94066.log.gz/var/log/ceph/cluster1-client.mirror.92555.log: 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129770.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123952.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.129770.log: -- /var/log/ceph/cluster2-client.mirror.120009.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129770.log.gz 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99659.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.120009.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117661.log.gz 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124684.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99659.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120009.log.gz 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123952.log.gz 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99659.log.gz 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101280.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112458.log 2023-12-02T23:03:13.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118935.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101280.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112458.log: gzip -5 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124684.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.106597.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118935.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112458.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122287.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.106597.log: -- replaced with /var/log/ceph/cluster2-client.mirror.118935.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125357.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125377.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122287.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106597.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125357.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117537.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122287.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125377.log: gzip -5 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125377.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125357.log.gz 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.107346.log 2023-12-02T23:03:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.117537.log: /var/log/ceph/cluster2-client.mirror.92793.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130022.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92793.log: 89.7% 75.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92793.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.107346.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117537.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95218.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.92555.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130022.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130022.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129013.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.95218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92700.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0%/var/log/ceph/cluster2-client.admin.129013.log: -- replaced with /var/log/ceph/cluster1-client.admin.95218.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129013.log.gz 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119224.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95380.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101974.log 2023-12-02T23:03:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95380.log: /var/log/ceph/cluster2-client.mirror.119224.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.95380.log.gz 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100353.log 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119224.log.gz 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111021.log 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101974.log.gz 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117971.log 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111021.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100353.log.gz 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111021.log.gz 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101897.log 2023-12-02T23:03:13.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107217.log 2023-12-02T23:03:13.252 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101897.log: /var/log/ceph/cluster2-client.mirror.117971.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117971.log.gz 2023-12-02T23:03:13.252 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130820.log 2023-12-02T23:03:13.252 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.92700.log.gz 2023-12-02T23:03:13.252 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107217.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107217.log.gz 2023-12-02T23:03:13.252 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102154.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101418.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102154.log: /var/log/ceph/cluster2-client.admin.130820.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130820.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110708.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101418.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101418.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130400.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110708.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110708.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98427.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130400.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130400.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112086.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98427.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98427.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89293.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112086.log: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112086.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123570.log 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89293.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89293.log.gz 2023-12-02T23:03:13.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129476.log 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123570.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123570.log.gz 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98467.log 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129476.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129476.log.gz 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102818.log 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98467.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98467.log.gz 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109933.log 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102818.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102818.log.gz 2023-12-02T23:03:13.254 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94662.log 2023-12-02T23:03:13.255 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109933.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109933.log.gz 2023-12-02T23:03:13.255 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118199.log 2023-12-02T23:03:13.268 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107462.log 2023-12-02T23:03:13.268 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118199.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118199.log.gz 2023-12-02T23:03:13.284 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100113.log 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107462.log: 87.6% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99859.log.gz 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112731.log 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107462.log.gz 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100113.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100113.log.gz 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89458.log 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112731.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112731.log.gz 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126822.log 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89458.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89458.log.gz 2023-12-02T23:03:13.291 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121484.log 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126822.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126822.log.gz 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122197.log 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121484.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121484.log.gz 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124705.log 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122197.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122197.log.gz 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110375.log 2023-12-02T23:03:13.296 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124705.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124705.log.gz 2023-12-02T23:03:13.297 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128929.log 2023-12-02T23:03:13.299 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110375.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110375.log.gz 2023-12-02T23:03:13.299 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118095.log 2023-12-02T23:03:13.300 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128929.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128929.log.gz 2023-12-02T23:03:13.301 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115641.log 2023-12-02T23:03:13.303 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118095.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118095.log.gz 2023-12-02T23:03:13.303 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106575.log 2023-12-02T23:03:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115641.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115641.log.gz 2023-12-02T23:03:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92345.log 2023-12-02T23:03:13.307 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106575.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106575.log.gz 2023-12-02T23:03:13.307 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97006.log 2023-12-02T23:03:13.308 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95996.log 2023-12-02T23:03:13.309 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97006.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92345.log.gz 2023-12-02T23:03:13.315 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97006.log.gz 2023-12-02T23:03:13.315 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98954.log 2023-12-02T23:03:13.321 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95996.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95996.log.gz 2023-12-02T23:03:13.321 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107503.log 2023-12-02T23:03:13.323 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98954.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98954.log.gz 2023-12-02T23:03:13.323 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105361.log 2023-12-02T23:03:13.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107503.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107503.log.gz 2023-12-02T23:03:13.325 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121236.log 2023-12-02T23:03:13.326 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105361.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105361.log.gz 2023-12-02T23:03:13.329 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125234.log 2023-12-02T23:03:13.329 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121236.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121236.log.gz 2023-12-02T23:03:13.329 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101438.log 2023-12-02T23:03:13.329 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125234.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125234.log.gz 2023-12-02T23:03:13.329 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99221.log 2023-12-02T23:03:13.331 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101438.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101438.log.gz 2023-12-02T23:03:13.331 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110925.log 2023-12-02T23:03:13.331 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99221.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99221.log.gz 2023-12-02T23:03:13.331 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98407.log 2023-12-02T23:03:13.332 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110925.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110925.log.gz 2023-12-02T23:03:13.333 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106141.log 2023-12-02T23:03:13.334 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98407.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98407.log.gz 2023-12-02T23:03:13.334 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111578.log 2023-12-02T23:03:13.337 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106141.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106141.log.gz 2023-12-02T23:03:13.337 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100992.log 2023-12-02T23:03:13.338 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111578.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111578.log.gz 2023-12-02T23:03:13.338 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94330.log 2023-12-02T23:03:13.352 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126188.log 2023-12-02T23:03:13.352 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94330.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94330.log.gz 2023-12-02T23:03:13.366 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99555.log 2023-12-02T23:03:13.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119059.log 2023-12-02T23:03:13.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99555.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99555.log.gz 2023-12-02T23:03:13.394 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89923.log 2023-12-02T23:03:13.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119059.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130442.log 2023-12-02T23:03:13.403 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119059.log.gz 2023-12-02T23:03:13.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89923.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89923.log.gz 2023-12-02T23:03:13.423 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91255.log 2023-12-02T23:03:13.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130442.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130442.log.gz 2023-12-02T23:03:13.423 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.125132.log.gz 2023-12-02T23:03:13.423 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121461.log 2023-12-02T23:03:13.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.91255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105847.log 2023-12-02T23:03:13.423 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121461.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121461.log.gz 2023-12-02T23:03:13.425 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106121.log 2023-12-02T23:03:13.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105847.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105847.log.gz 2023-12-02T23:03:13.425 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127375.log 2023-12-02T23:03:13.438 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106121.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106121.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121146.log 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127375.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127375.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127225.log 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121146.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121146.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90133.log 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127225.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127225.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111890.log 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.90133.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90133.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124278.log 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111890.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111890.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 80.1% -5 -- replaced with /var/log/ceph/cluster2-client.admin.91255.log.gz 2023-12-02T23:03:13.439 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.126146.log 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124278.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124278.log.gz 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119617.log 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126146.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126146.log.gz 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112948.log 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119617.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119617.log.gz 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128341.log 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112948.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112948.log.gz 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117827.log 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128341.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128341.log.gz 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97826.log 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117827.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117827.log.gz 2023-12-02T23:03:13.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117723.log 2023-12-02T23:03:13.443 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97826.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97826.log.gz 2023-12-02T23:03:13.444 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105888.log 2023-12-02T23:03:13.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117723.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117723.log.gz 2023-12-02T23:03:13.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92664.log 2023-12-02T23:03:13.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105888.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105888.log.gz 2023-12-02T23:03:13.448 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100029.log 2023-12-02T23:03:13.449 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105448.log 2023-12-02T23:03:13.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103318.log 2023-12-02T23:03:13.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105448.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105448.log.gz 2023-12-02T23:03:13.474 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112261.log 2023-12-02T23:03:13.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103318.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103318.log.gz 2023-12-02T23:03:13.488 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117558.log 2023-12-02T23:03:13.489 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112261.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112261.log.gz 2023-12-02T23:03:13.497 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118054.log 2023-12-02T23:03:13.497 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117558.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117558.log.gz 2023-12-02T23:03:13.522 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122062.log 2023-12-02T23:03:13.523 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118054.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118054.log.gz 2023-12-02T23:03:13.534 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126651.log.gz 2023-12-02T23:03:13.536 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115280.log 2023-12-02T23:03:13.537 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122062.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122062.log.gz 2023-12-02T23:03:13.551 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125399.log.gz 2023-12-02T23:03:13.551 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90175.log 2023-12-02T23:03:13.553 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115280.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115280.log.gz 2023-12-02T23:03:13.553 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122653.log 2023-12-02T23:03:13.553 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.90175.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90175.log.gz 2023-12-02T23:03:13.562 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112434.log 2023-12-02T23:03:13.563 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.122653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116323.log 2023-12-02T23:03:13.563 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112434.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112434.log.gz 2023-12-02T23:03:13.582 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103786.log 2023-12-02T23:03:13.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116323.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116323.log.gz 2023-12-02T23:03:13.588 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93767.log.gz 2023-12-02T23:03:13.590 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97505.log 2023-12-02T23:03:13.603 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103786.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103786.log.gz 2023-12-02T23:03:13.612 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-02T23:03:13.616 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125958.log 2023-12-02T23:03:13.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97505.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104288.log 2023-12-02T23:03:13.635 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125958.log: -- replaced with /var/log/ceph/cluster2-client.mirror.97505.log.gz 2023-12-02T23:03:13.635 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125958.log.gz 2023-12-02T23:03:13.635 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.1.100696.log 2023-12-02T23:03:13.637 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104288.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103808.log.gz 2023-12-02T23:03:13.646 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113793.log 2023-12-02T23:03:13.647 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.1.100696.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104288.log.gz 2023-12-02T23:03:13.656 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102154.log.gz 2023-12-02T23:03:13.659 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91169.log 2023-12-02T23:03:13.660 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113793.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113793.log.gz 2023-12-02T23:03:13.672 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90747.log 2023-12-02T23:03:13.674 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91169.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91169.log.gz 2023-12-02T23:03:13.677 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108379.log 2023-12-02T23:03:13.683 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90747.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90747.log.gz 2023-12-02T23:03:13.690 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123990.log 2023-12-02T23:03:13.691 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108379.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108379.log.gz 2023-12-02T23:03:13.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108708.log 2023-12-02T23:03:13.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123990.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123990.log.gz 2023-12-02T23:03:13.718 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100543.log 2023-12-02T23:03:13.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108708.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108708.log.gz 2023-12-02T23:03:13.726 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119865.log 2023-12-02T23:03:13.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110178.log 2023-12-02T23:03:13.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119865.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100992.log.gz 2023-12-02T23:03:13.736 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116909.log 2023-12-02T23:03:13.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110178.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110178.log.gz 2023-12-02T23:03:13.738 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92756.log 2023-12-02T23:03:13.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116909.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116909.log.gz 2023-12-02T23:03:13.740 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130274.log 2023-12-02T23:03:13.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126046.log 2023-12-02T23:03:13.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130274.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130274.log.gz 2023-12-02T23:03:13.741 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112213.log 2023-12-02T23:03:13.741 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92756.log.gz 2023-12-02T23:03:13.742 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126046.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126046.log.gz 2023-12-02T23:03:13.742 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110642.log 2023-12-02T23:03:13.743 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112213.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112213.log.gz 2023-12-02T23:03:13.743 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125420.log 2023-12-02T23:03:13.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110642.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110642.log.gz 2023-12-02T23:03:13.745 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106964.log 2023-12-02T23:03:13.745 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125420.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125420.log.gz 2023-12-02T23:03:13.746 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120561.log 2023-12-02T23:03:13.746 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106964.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106964.log.gz 2023-12-02T23:03:13.746 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90007.log 2023-12-02T23:03:13.747 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120561.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120561.log.gz 2023-12-02T23:03:13.749 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111303.log 2023-12-02T23:03:13.749 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119865.log.gz 2023-12-02T23:03:13.749 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.90007.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90007.log.gz 2023-12-02T23:03:13.749 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97611.log 2023-12-02T23:03:13.751 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111303.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111303.log.gz 2023-12-02T23:03:13.751 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91340.log 2023-12-02T23:03:13.758 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97611.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97611.log.gz 2023-12-02T23:03:13.758 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89965.log 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101079.log 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91340.log.gz 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89965.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89965.log.gz 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89151.log 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101079.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101079.log.gz 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105486.log 2023-12-02T23:03:13.762 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89151.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89151.log.gz 2023-12-02T23:03:13.763 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123591.log 2023-12-02T23:03:13.765 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105486.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105486.log.gz 2023-12-02T23:03:13.765 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110238.log 2023-12-02T23:03:13.765 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123591.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123591.log.gz 2023-12-02T23:03:13.765 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120719.log 2023-12-02T23:03:13.768 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110238.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110238.log.gz 2023-12-02T23:03:13.768 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.126225.log 2023-12-02T23:03:13.768 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120719.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120719.log.gz 2023-12-02T23:03:13.768 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108403.log 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.126225.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100029.log.gz 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93223.log 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108403.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108403.log.gz 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.126225.log.gz 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130568.log 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107631.log 2023-12-02T23:03:13.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130568.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93223.log.gz 2023-12-02T23:03:13.775 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130568.log.gz 2023-12-02T23:03:13.775 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116423.log 2023-12-02T23:03:13.775 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107631.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107631.log.gz 2023-12-02T23:03:13.775 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112856.log 2023-12-02T23:03:13.780 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116423.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116423.log.gz 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118976.log 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112856.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112856.log.gz 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127921.log 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118976.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118976.log.gz 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101339.log 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127921.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127921.log.gz 2023-12-02T23:03:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95614.log 2023-12-02T23:03:13.783 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101339.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101339.log.gz 2023-12-02T23:03:13.784 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97177.log 2023-12-02T23:03:13.784 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.95614.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.95614.log.gz 2023-12-02T23:03:13.784 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105426.log 2023-12-02T23:03:13.784 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97177.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.97177.log.gz 2023-12-02T23:03:13.785 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.109646.log 2023-12-02T23:03:13.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105426.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105426.log.gz 2023-12-02T23:03:13.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108073.log 2023-12-02T23:03:13.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.3.109646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95066.log 2023-12-02T23:03:13.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108073.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108073.log.gz 2023-12-02T23:03:13.808 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114986.log 2023-12-02T23:03:13.821 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126760.log 2023-12-02T23:03:13.821 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95066.log.gz 2023-12-02T23:03:13.821 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114986.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114986.log.gz 2023-12-02T23:03:13.830 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96463.log 2023-12-02T23:03:13.830 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126760.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126760.log.gz 2023-12-02T23:03:13.842 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98041.log 2023-12-02T23:03:13.843 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96463.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96463.log.gz 2023-12-02T23:03:13.850 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122653.log.gz 2023-12-02T23:03:13.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104936.log 2023-12-02T23:03:13.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98041.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98041.log.gz 2023-12-02T23:03:13.862 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104960.log 2023-12-02T23:03:13.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104936.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.104936.log.gz 2023-12-02T23:03:13.862 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122085.log 2023-12-02T23:03:13.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104960.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.104960.log.gz 2023-12-02T23:03:13.862 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.96531.log 2023-12-02T23:03:13.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122085.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122085.log.gz 2023-12-02T23:03:13.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125299.log 2023-12-02T23:03:13.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.2.96531.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.109646.log.gz 2023-12-02T23:03:13.872 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.96531.log.gz 2023-12-02T23:03:13.872 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104384.log 2023-12-02T23:03:13.872 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123376.log 2023-12-02T23:03:13.872 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125299.log.gz/var/log/ceph/cluster1-client.mirror.104384.log: 2023-12-02T23:03:13.872 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.104384.log.gz 2023-12-02T23:03:13.872 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115073.log 2023-12-02T23:03:13.873 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96625.log 2023-12-02T23:03:13.873 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123376.log.gz/var/log/ceph/cluster2-client.mirror.115073.log: 2023-12-02T23:03:13.873 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124477.log 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.96625.log: 89.8% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115073.log.gz 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.96625.log.gz 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109224.log 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.124477.log: -- /var/log/ceph/cluster1-client.mirror.91118.log 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124477.log.gz 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109606.log 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110463.log 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109606.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109224.log.gz 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.90524.log 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110463.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109606.log.gz 2023-12-02T23:03:13.883 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110463.log.gz 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105385.log 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97912.log 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105385.log: /var/log/ceph/cluster1-client.mirror.1.90524.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105385.log.gz 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.127107.log 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.97912.log: -- /var/log/ceph/cluster2-client.mirror.115617.log 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.127107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127107.log.gz 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125697.log 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115617.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115617.log.gz 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114102.log 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125697.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125697.log.gz 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr: 54.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91118.log.gz 2023-12-02T23:03:13.884 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117103.log 2023-12-02T23:03:13.960 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114102.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114102.log.gz 2023-12-02T23:03:13.960 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115942.log 2023-12-02T23:03:13.960 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97697.log 2023-12-02T23:03:13.960 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97912.log.gz 2023-12-02T23:03:13.960 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117103.log: /var/log/ceph/cluster1-client.mirror.115942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119823.log 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97697.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115942.log.gz 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119162.log 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119823.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101897.log.gz 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120696.log 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119162.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119162.log.gz 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110258.log 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117103.log.gz 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115734.log 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119823.log.gz 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120696.log: gzip -5 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120696.log.gz 2023-12-02T23:03:13.961 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.122107.log 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115734.log: /var/log/ceph/cluster2-client.mirror.110258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114189.log 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122107.log: 87.3% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122107.log.gz 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.110258.log.gz 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115139.log 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114189.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95835.log 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97697.log.gz 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115734.log.gz 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114789.log 2023-12-02T23:03:13.962 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114189.log.gz 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103219.log 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115139.log.gz/var/log/ceph/cluster1-client.mirror.114789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115505.log 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95835.log.gz 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89457.log 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103219.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114789.log.gz 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.126396.log 2023-12-02T23:03:13.963 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115505.log.gz 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130610.log 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89457.log.gz 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.3.126396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90524.log 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107856.log 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90524.log: /var/log/ceph/cluster2-client.admin.130610.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130610.log.gz 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101201.log 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90524.log.gz 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.107856.log: /var/log/ceph/cluster2-client.mirror.120092.log 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107856.log.gz 2023-12-02T23:03:13.964 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101201.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.126396.log.gz 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130526.log 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.120092.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103907.log 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.101201.log.gz 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130526.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130526.log.gz 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120092.log.gz 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103966.log 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.123457.log 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103907.log: /var/log/ceph/cluster2-client.mirror.103966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105578.log 2023-12-02T23:03:13.965 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.2.123457.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103907.log.gz 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118323.log 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105578.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105578.log.gz 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102840.log 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118323.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118323.log.gz 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124107.log 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102840.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.100696.log.gz 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102290.log 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124107.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124107.log.gz 2023-12-02T23:03:13.966 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102253.log 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102290.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102290.log.gz 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106052.log 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102253.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.102253.log.gz 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90343.log 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106052.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.123457.log.gz 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106052.log.gz 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101243.log 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.90343.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90343.log.gz 2023-12-02T23:03:13.967 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94832.log 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101243.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101243.log.gz 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.90524.log.gz 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98635.log 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.94832.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.94832.log.gz 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114388.log 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96739.log 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114388.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98635.log.gz 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114388.log.gz 2023-12-02T23:03:13.968 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118852.log 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118997.log 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96739.log: /var/log/ceph/cluster2-client.mirror.118852.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96739.log.gz 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118852.log.gz 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.96811.log 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127032.log 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96811.log: /var/log/ceph/cluster2-client.mirror.118997.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.96811.log.gz 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118997.log.gz 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106784.log 2023-12-02T23:03:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.127032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127032.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121056.log 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119989.log 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121056.log: /var/log/ceph/cluster2-client.mirror.106784.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121056.log.gz 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126716.log 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119989.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119989.log.gz 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92253.log 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126716.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106784.log.gz 2023-12-02T23:03:13.970 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92111.log 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92253.log: gzip 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126716.log.gz 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113394.log 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.92111.log: --verbose -- /var/log/ceph/cluster2-client.mirror.94414.log 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122535.log 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113394.log: /var/log/ceph/cluster2-client.mirror.94414.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113394.log.gz 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124436.log 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122535.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122535.log.gz 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92253.log.gz 2023-12-02T23:03:13.971 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94414.log.gz 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.117806.log 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110395.log 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117806.log: /var/log/ceph/cluster2-client.mirror.124436.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124436.log.gz 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102095.log 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110395.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117806.log.gz 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118219.log 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102095.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102095.log.gz 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110395.log.gz 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109119.log 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110354.log 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109119.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109119.log.gz 2023-12-02T23:03:13.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97763.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110354.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118219.log.gz 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110354.log.gz 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119720.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93168.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97763.log: /var/log/ceph/cluster2-client.mirror.119720.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119720.log.gz 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120674.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107969.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120674.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120674.log.gz 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127585.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107969.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107969.log.gz 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110282.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127585.log: 93.4% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127585.log.gz 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.97763.log.gz 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114717.log 2023-12-02T23:03:13.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.92253.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110282.log.gz 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114717.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94662.log.gz 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114717.log.gz 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.2.92253.log: -- /var/log/ceph/cluster2-client.mirror.103514.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125319.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102033.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125319.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103514.log.gz 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97419.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108468.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97419.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102033.log.gz 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118467.log 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125319.log.gz 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108468.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108468.log.gz 2023-12-02T23:03:13.974 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106486.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101590.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106486.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106486.log.gz 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102016.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101590.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97419.log.gz 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101590.log.gz 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118467.log.gz 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116852.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102016.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102016.log.gz 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89671.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106028.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89671.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116852.log.gz 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109888.log 2023-12-02T23:03:13.975 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.89671.log.gz 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.94702.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116539.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109888.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106028.log.gz 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.3.94702.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109888.log.gz 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122576.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116539.log.gz/var/log/ceph/cluster2-client.mirror.122576.log: 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122576.log.gz 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103536.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114914.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109565.log 2023-12-02T23:03:13.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103536.log: /var/log/ceph/cluster2-client.mirror.114914.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114914.log.gz 2023-12-02T23:03:13.978 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118281.log 2023-12-02T23:03:13.978 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112414.log 2023-12-02T23:03:13.978 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109565.log.gz 2023-12-02T23:03:13.980 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118281.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118281.log.gz 2023-12-02T23:03:13.983 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111697.log 2023-12-02T23:03:13.984 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112414.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112414.log.gz 2023-12-02T23:03:13.984 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117847.log 2023-12-02T23:03:13.984 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111697.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111697.log.gz 2023-12-02T23:03:13.986 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99701.log 2023-12-02T23:03:13.989 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117847.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117847.log.gz 2023-12-02T23:03:13.990 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122445.log 2023-12-02T23:03:14.008 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99701.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.94702.log.gz 2023-12-02T23:03:14.009 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113507.log 2023-12-02T23:03:14.010 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122445.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122445.log.gz 2023-12-02T23:03:14.010 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117061.log 2023-12-02T23:03:14.012 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113507.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113507.log.gz 2023-12-02T23:03:14.013 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90427.log 2023-12-02T23:03:14.014 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117061.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117061.log.gz 2023-12-02T23:03:14.015 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93016.log 2023-12-02T23:03:14.015 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.90427.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90427.log.gz 2023-12-02T23:03:14.015 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117765.log 2023-12-02T23:03:14.016 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102719.log 2023-12-02T23:03:14.016 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117765.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117765.log.gz 2023-12-02T23:03:14.017 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102396.log 2023-12-02T23:03:14.018 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102719.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93016.log.gz 2023-12-02T23:03:14.018 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102719.log.gz 2023-12-02T23:03:14.019 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104425.log 2023-12-02T23:03:14.020 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102396.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102396.log.gz 2023-12-02T23:03:14.020 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85729.log 2023-12-02T23:03:14.023 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104425.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104425.log.gz 2023-12-02T23:03:14.024 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104864.log 2023-12-02T23:03:14.024 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.85729.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85729.log.gz 2023-12-02T23:03:14.024 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130232.log 2023-12-02T23:03:14.026 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104864.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.104864.log.gz 2023-12-02T23:03:14.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126609.log 2023-12-02T23:03:14.026 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130232.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130232.log.gz 2023-12-02T23:03:14.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97849.log 2023-12-02T23:03:14.027 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126609.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126609.log.gz 2023-12-02T23:03:14.028 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91306.log 2023-12-02T23:03:14.036 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97849.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97849.log.gz 2023-12-02T23:03:14.037 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93437.log 2023-12-02T23:03:14.037 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118385.log 2023-12-02T23:03:14.037 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106804.log 2023-12-02T23:03:14.037 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118385.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118385.log.gz 2023-12-02T23:03:14.040 INFO:teuthology.orchestra.run.smithi170.stderr: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91306.log.gz 2023-12-02T23:03:14.043 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93437.log.gz 2023-12-02T23:03:14.044 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106940.log 2023-12-02T23:03:14.050 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106804.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106804.log.gz 2023-12-02T23:03:14.050 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110158.log 2023-12-02T23:03:14.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106940.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106940.log.gz 2023-12-02T23:03:14.054 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113073.log 2023-12-02T23:03:14.058 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110158.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.110158.log.gz 2023-12-02T23:03:14.059 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98267.log 2023-12-02T23:03:14.062 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113073.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126188.log.gz 2023-12-02T23:03:14.063 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111536.log 2023-12-02T23:03:14.064 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113073.log.gz 2023-12-02T23:03:14.064 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92968.log 2023-12-02T23:03:14.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111536.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98267.log.gz 2023-12-02T23:03:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95194.log 2023-12-02T23:03:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96441.log 2023-12-02T23:03:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95194.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.92968.log.gz 2023-12-02T23:03:14.069 INFO:teuthology.orchestra.run.smithi170.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111536.log.gz 2023-12-02T23:03:14.069 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121326.log 2023-12-02T23:03:14.078 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96441.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96441.log.gz 2023-12-02T23:03:14.078 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91323.log 2023-12-02T23:03:14.079 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121326.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121326.log.gz 2023-12-02T23:03:14.079 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105799.log 2023-12-02T23:03:14.089 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99258.log 2023-12-02T23:03:14.090 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105799.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105799.log.gz 2023-12-02T23:03:14.094 INFO:teuthology.orchestra.run.smithi170.stderr: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91323.log.gz 2023-12-02T23:03:14.097 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128047.log 2023-12-02T23:03:14.103 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99258.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99258.log.gz 2023-12-02T23:03:14.103 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104312.log 2023-12-02T23:03:14.107 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128047.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128047.log.gz 2023-12-02T23:03:14.107 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93081.log 2023-12-02T23:03:14.112 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104312.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.104312.log.gz 2023-12-02T23:03:14.112 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129518.log 2023-12-02T23:03:14.112 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89585.log 2023-12-02T23:03:14.112 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129518.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129518.log.gz 2023-12-02T23:03:14.112 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112285.log 2023-12-02T23:03:14.116 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89585.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89585.log.gz 2023-12-02T23:03:14.116 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94462.log 2023-12-02T23:03:14.117 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112285.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112285.log.gz 2023-12-02T23:03:14.122 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130106.log 2023-12-02T23:03:14.122 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94462.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94462.log.gz 2023-12-02T23:03:14.122 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126225.log 2023-12-02T23:03:14.122 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130106.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130106.log.gz 2023-12-02T23:03:14.122 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.93286.log 2023-12-02T23:03:14.131 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126225.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126225.log.gz 2023-12-02T23:03:14.131 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115547.log 2023-12-02T23:03:14.131 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.93286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118426.log 2023-12-02T23:03:14.137 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115547.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115547.log.gz 2023-12-02T23:03:14.140 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107282.log 2023-12-02T23:03:14.148 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118426.log: 89.3% 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93081.log.gz 2023-12-02T23:03:14.148 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118426.log.gz 2023-12-02T23:03:14.149 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114251.log 2023-12-02T23:03:14.154 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107282.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107282.log.gz 2023-12-02T23:03:14.154 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103699.log 2023-12-02T23:03:14.158 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114251.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114251.log.gz 2023-12-02T23:03:14.158 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119079.log 2023-12-02T23:03:14.162 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103699.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103699.log.gz 2023-12-02T23:03:14.162 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109268.log 2023-12-02T23:03:14.172 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115963.log 2023-12-02T23:03:14.174 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109268.log: 87.7% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119079.log.gz 2023-12-02T23:03:14.174 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109268.log.gz 2023-12-02T23:03:14.177 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107053.log 2023-12-02T23:03:14.185 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115963.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115963.log.gz 2023-12-02T23:03:14.185 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122220.log 2023-12-02T23:03:14.189 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107053.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103536.log.gz 2023-12-02T23:03:14.190 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107053.log.gz 2023-12-02T23:03:14.191 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93953.log 2023-12-02T23:03:14.192 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122220.log: 93.8% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122220.log.gz 2023-12-02T23:03:14.192 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100543.log.gz 2023-12-02T23:03:14.192 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113213.log 2023-12-02T23:03:14.193 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124494.log 2023-12-02T23:03:14.194 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113213.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93953.log.gz 2023-12-02T23:03:14.194 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102497.log 2023-12-02T23:03:14.194 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113213.log.gz 2023-12-02T23:03:14.195 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124494.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124494.log.gz 2023-12-02T23:03:14.199 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109541.log 2023-12-02T23:03:14.199 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115846.log 2023-12-02T23:03:14.199 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130694.log 2023-12-02T23:03:14.200 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109541.log.gz 2023-12-02T23:03:14.200 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97439.log 2023-12-02T23:03:14.200 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130694.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130694.log.gz 2023-12-02T23:03:14.200 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115846.log.gz 2023-12-02T23:03:14.200 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98021.log 2023-12-02T23:03:14.200 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94001.log 2023-12-02T23:03:14.211 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98021.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98021.log.gz 2023-12-02T23:03:14.211 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129308.log 2023-12-02T23:03:14.211 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116515.log 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129308.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96247.log 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.93286.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116515.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116515.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98508.log 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96247.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103966.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94001.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98508.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.96247.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103118.log 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98508.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124514.log 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103118.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103118.log.gz 2023-12-02T23:03:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124728.log 2023-12-02T23:03:14.213 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98971.log 2023-12-02T23:03:14.213 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124514.log.gz 2023-12-02T23:03:14.214 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97439.log.gz 2023-12-02T23:03:14.214 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98971.log: /var/log/ceph/cluster1-client.mirror.124728.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124728.log.gz 2023-12-02T23:03:14.214 INFO:teuthology.orchestra.run.smithi170.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98971.log.gz 2023-12-02T23:03:14.218 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114147.log 2023-12-02T23:03:14.218 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119431.log 2023-12-02T23:03:14.218 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94575.log 2023-12-02T23:03:14.218 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129854.log 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119431.log.gz/var/log/ceph/cluster1-client.mirror.94575.log: 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128761.log 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114147.log.gz/var/log/ceph/cluster2-client.admin.129854.log: 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129854.log.gz 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128761.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106647.log 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.128761.log.gz 2023-12-02T23:03:14.219 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117785.log 2023-12-02T23:03:14.220 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106647.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106647.log.gz 2023-12-02T23:03:14.220 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111325.log 2023-12-02T23:03:14.222 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117785.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117785.log.gz 2023-12-02T23:03:14.223 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94575.log.gz 2023-12-02T23:03:14.223 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111628.log 2023-12-02T23:03:14.223 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116668.log 2023-12-02T23:03:14.226 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111628.log: /var/log/ceph/cluster2-client.mirror.111325.log: 89.7% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111325.log.gz 2023-12-02T23:03:14.226 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.111628.log.gz 2023-12-02T23:03:14.226 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114693.log 2023-12-02T23:03:14.226 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95312.log 2023-12-02T23:03:14.226 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116668.log.gz 2023-12-02T23:03:14.227 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96719.log 2023-12-02T23:03:14.227 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114589.log 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114693.log.gz 2023-12-02T23:03:14.227 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96719.log: /var/log/ceph/cluster2-client.mirror.95312.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.96719.log.gz 2023-12-02T23:03:14.227 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118509.log 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95312.log.gz 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119038.log 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118509.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118509.log.gz 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127153.log 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102233.log 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119038.log.gz 2023-12-02T23:03:14.233 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114589.log.gz 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111986.log 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.127153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.127153.log.gz 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128845.log 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111986.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111986.log.gz 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111166.log 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128845.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128845.log.gz 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102534.log 2023-12-02T23:03:14.234 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111166.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111166.log.gz 2023-12-02T23:03:14.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102191.log 2023-12-02T23:03:14.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102534.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102534.log.gz 2023-12-02T23:03:14.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113880.log 2023-12-02T23:03:14.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102191.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102191.log.gz 2023-12-02T23:03:14.237 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124130.log 2023-12-02T23:03:14.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113880.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113880.log.gz 2023-12-02T23:03:14.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91152.log 2023-12-02T23:03:14.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124130.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124130.log.gz 2023-12-02T23:03:14.241 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109493.log 2023-12-02T23:03:14.241 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96485.log 2023-12-02T23:03:14.242 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109493.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109493.log.gz 2023-12-02T23:03:14.242 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113817.log 2023-12-02T23:03:14.249 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96485.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91152.log.gz 2023-12-02T23:03:14.252 INFO:teuthology.orchestra.run.smithi170.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.92253.log.gz 2023-12-02T23:03:14.260 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92944.log 2023-12-02T23:03:14.261 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113817.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113817.log.gz 2023-12-02T23:03:14.262 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110579.log 2023-12-02T23:03:14.263 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122400.log 2023-12-02T23:03:14.264 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110579.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110579.log.gz 2023-12-02T23:03:14.264 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102075.log 2023-12-02T23:03:14.264 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122400.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122400.log.gz 2023-12-02T23:03:14.265 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100563.log 2023-12-02T23:03:14.274 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102075.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99701.log.gz 2023-12-02T23:03:14.275 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128257.log 2023-12-02T23:03:14.276 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.92944.log.gz 2023-12-02T23:03:14.277 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100563.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100563.log.gz 2023-12-02T23:03:14.277 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118261.log 2023-12-02T23:03:14.277 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128257.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128257.log.gz 2023-12-02T23:03:14.291 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97310.log 2023-12-02T23:03:14.292 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118261.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118261.log.gz 2023-12-02T23:03:14.292 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126781.log 2023-12-02T23:03:14.298 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97310.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97310.log.gz 2023-12-02T23:03:14.298 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120606.log 2023-12-02T23:03:14.299 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126781.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126781.log.gz 2023-12-02T23:03:14.300 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115163.log 2023-12-02T23:03:14.301 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120606.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120606.log.gz 2023-12-02T23:03:14.301 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125090.log 2023-12-02T23:03:14.301 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115163.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115163.log.gz 2023-12-02T23:03:14.302 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97892.log 2023-12-02T23:03:14.304 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125090.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125090.log.gz 2023-12-02T23:03:14.304 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104447.log 2023-12-02T23:03:14.310 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97892.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97892.log.gz 2023-12-02T23:03:14.311 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-02T23:03:14.311 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104447.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104447.log.gz 2023-12-02T23:03:14.312 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114231.log 2023-12-02T23:03:14.318 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2.log: 87.9% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-02T23:03:14.318 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122611.log 2023-12-02T23:03:14.320 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114231.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114231.log.gz 2023-12-02T23:03:14.320 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123632.log 2023-12-02T23:03:14.322 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.122611.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122611.log.gz 2023-12-02T23:03:14.322 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110065.log 2023-12-02T23:03:14.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123632.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123632.log.gz 2023-12-02T23:03:14.325 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96324.log 2023-12-02T23:03:14.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110065.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110065.log.gz 2023-12-02T23:03:14.326 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107808.log 2023-12-02T23:03:14.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96324.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.96324.log.gz 2023-12-02T23:03:14.331 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103887.log 2023-12-02T23:03:14.331 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107808.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107808.log.gz 2023-12-02T23:03:14.331 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90301.log 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103887.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103219.log.gz 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98307.log 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.90301.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90301.log.gz 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105001.log 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98307.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98307.log.gz 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117083.log 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105001.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105001.log.gz 2023-12-02T23:03:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92187.log 2023-12-02T23:03:14.347 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117083.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117083.log.gz 2023-12-02T23:03:14.347 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114126.log 2023-12-02T23:03:14.347 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117351.log 2023-12-02T23:03:14.347 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114126.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.92187.log.gz 2023-12-02T23:03:14.347 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114126.log.gz 2023-12-02T23:03:14.347 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100659.log 2023-12-02T23:03:14.349 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117351.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117351.log.gz 2023-12-02T23:03:14.350 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106667.log 2023-12-02T23:03:14.356 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100659.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100659.log.gz 2023-12-02T23:03:14.356 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116648.log 2023-12-02T23:03:14.356 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106667.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106667.log.gz 2023-12-02T23:03:14.356 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115211.log 2023-12-02T23:03:14.357 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116648.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116648.log.gz 2023-12-02T23:03:14.357 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93286.log 2023-12-02T23:03:14.357 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115211.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115211.log.gz 2023-12-02T23:03:14.357 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98127.log 2023-12-02T23:03:14.357 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124032.log 2023-12-02T23:03:14.358 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98127.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98127.log.gz 2023-12-02T23:03:14.358 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94046.log 2023-12-02T23:03:14.360 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124032.log: 81.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93286.log.gz 2023-12-02T23:03:14.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122512.log 2023-12-02T23:03:14.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99112.log 2023-12-02T23:03:14.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122512.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122512.log.gz 2023-12-02T23:03:14.395 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108666.log 2023-12-02T23:03:14.396 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99112.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99112.log.gz 2023-12-02T23:03:14.404 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94046.log.gz 2023-12-02T23:03:14.413 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104770.log 2023-12-02T23:03:14.414 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108666.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108666.log.gz 2023-12-02T23:03:14.429 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89837.log 2023-12-02T23:03:14.429 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104770.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104770.log.gz 2023-12-02T23:03:14.437 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102840.log.gz 2023-12-02T23:03:14.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109361.log 2023-12-02T23:03:14.442 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89837.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89837.log.gz 2023-12-02T23:03:14.442 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99891.log 2023-12-02T23:03:14.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109361.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109361.log.gz 2023-12-02T23:03:14.448 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102576.log 2023-12-02T23:03:14.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99891.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99891.log.gz 2023-12-02T23:03:14.449 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108621.log 2023-12-02T23:03:14.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113189.log 2023-12-02T23:03:14.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108621.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108621.log.gz 2023-12-02T23:03:14.487 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104268.log 2023-12-02T23:03:14.487 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113189.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113189.log.gz 2023-12-02T23:03:14.496 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111769.log 2023-12-02T23:03:14.505 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91430.log 2023-12-02T23:03:14.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111769.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95194.log.gz 2023-12-02T23:03:14.510 INFO:teuthology.orchestra.run.smithi170.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104268.log.gz 2023-12-02T23:03:14.510 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111769.log.gz 2023-12-02T23:03:14.513 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126509.log 2023-12-02T23:03:14.515 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96203.log 2023-12-02T23:03:14.515 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126509.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126509.log.gz 2023-12-02T23:03:14.515 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124370.log 2023-12-02T23:03:14.516 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96203.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91430.log.gz 2023-12-02T23:03:14.517 INFO:teuthology.orchestra.run.smithi170.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.96203.log.gz 2023-12-02T23:03:14.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92135.log 2023-12-02T23:03:14.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124370.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124370.log.gz 2023-12-02T23:03:14.520 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107169.log 2023-12-02T23:03:14.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116443.log 2023-12-02T23:03:14.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107169.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.107169.log.gz 2023-12-02T23:03:14.520 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94888.log 2023-12-02T23:03:14.521 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116443.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.92135.log.gz 2023-12-02T23:03:14.521 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116443.log.gz 2023-12-02T23:03:14.521 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93745.log 2023-12-02T23:03:14.524 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94888.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94888.log.gz 2023-12-02T23:03:14.524 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119782.log 2023-12-02T23:03:14.524 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113370.log 2023-12-02T23:03:14.524 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119782.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119782.log.gz 2023-12-02T23:03:14.525 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107928.log 2023-12-02T23:03:14.525 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113370.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113370.log.gz 2023-12-02T23:03:14.526 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97869.log 2023-12-02T23:03:14.526 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107928.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.107928.log.gz 2023-12-02T23:03:14.527 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120030.log 2023-12-02T23:03:14.532 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97869.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93745.log.gz 2023-12-02T23:03:14.533 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97869.log.gz 2023-12-02T23:03:14.534 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124967.log 2023-12-02T23:03:14.538 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.120030.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120030.log.gz 2023-12-02T23:03:14.539 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100076.log 2023-12-02T23:03:14.539 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124967.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.124967.log.gz 2023-12-02T23:03:14.539 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105910.log 2023-12-02T23:03:14.539 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100076.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.100076.log.gz 2023-12-02T23:03:14.539 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108778.log 2023-12-02T23:03:14.541 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105910.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105910.log.gz 2023-12-02T23:03:14.541 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110507.log 2023-12-02T23:03:14.541 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108778.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108778.log.gz 2023-12-02T23:03:14.541 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109403.log 2023-12-02T23:03:14.542 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110507.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110507.log.gz 2023-12-02T23:03:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114347.log 2023-12-02T23:03:14.544 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109403.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109403.log.gz 2023-12-02T23:03:14.544 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115525.log 2023-12-02T23:03:14.544 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114347.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114347.log.gz 2023-12-02T23:03:14.546 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101669.log 2023-12-02T23:03:14.548 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115525.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115525.log.gz 2023-12-02T23:03:14.548 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97225.log 2023-12-02T23:03:14.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.101669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108029.log 2023-12-02T23:03:14.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97225.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97225.log.gz 2023-12-02T23:03:14.555 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117040.log 2023-12-02T23:03:14.557 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108029.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108029.log.gz 2023-12-02T23:03:14.557 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104707.log 2023-12-02T23:03:14.560 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117040.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117040.log.gz 2023-12-02T23:03:14.560 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121169.log 2023-12-02T23:03:14.563 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104707.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.104707.log.gz 2023-12-02T23:03:14.563 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122265.log 2023-12-02T23:03:14.563 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121169.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121169.log.gz 2023-12-02T23:03:14.563 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114609.log 2023-12-02T23:03:14.564 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122265.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122265.log.gz 2023-12-02T23:03:14.567 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103494.log 2023-12-02T23:03:14.576 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93061.log 2023-12-02T23:03:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103494.log: 89.4% 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114609.log.gz 2023-12-02T23:03:14.580 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103494.log.gz 2023-12-02T23:03:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108251.log 2023-12-02T23:03:14.580 INFO:teuthology.orchestra.run.smithi170.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101669.log.gz 2023-12-02T23:03:14.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114167.log 2023-12-02T23:03:14.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108251.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108251.log.gz 2023-12-02T23:03:14.582 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95428.log 2023-12-02T23:03:14.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114167.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114167.log.gz 2023-12-02T23:03:14.585 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105289.log 2023-12-02T23:03:14.585 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95428.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.95428.log.gz 2023-12-02T23:03:14.586 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99642.log 2023-12-02T23:03:14.586 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105289.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105289.log.gz 2023-12-02T23:03:14.588 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113418.log 2023-12-02T23:03:14.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99642.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99642.log.gz 2023-12-02T23:03:14.588 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101260.log 2023-12-02T23:03:14.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113418.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113418.log.gz 2023-12-02T23:03:14.589 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127837.log 2023-12-02T23:03:14.590 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101260.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101260.log.gz 2023-12-02T23:03:14.590 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111914.log 2023-12-02T23:03:14.590 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127837.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127837.log.gz 2023-12-02T23:03:14.590 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117020.log 2023-12-02T23:03:14.594 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111914.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111914.log.gz 2023-12-02T23:03:14.595 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119927.log 2023-12-02T23:03:14.601 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103887.log.gz 2023-12-02T23:03:14.601 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117020.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117020.log.gz 2023-12-02T23:03:14.603 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105023.log 2023-12-02T23:03:14.605 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119927.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119927.log.gz 2023-12-02T23:03:14.605 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112350.log 2023-12-02T23:03:14.606 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121722.log 2023-12-02T23:03:14.606 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112350.log: /var/log/ceph/cluster2-client.mirror.105023.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105023.log.gz 2023-12-02T23:03:14.606 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101302.log 2023-12-02T23:03:14.607 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121722.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121722.log.gz 2023-12-02T23:03:14.607 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112350.log.gz 2023-12-02T23:03:14.607 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105506.log 2023-12-02T23:03:14.608 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127627.log 2023-12-02T23:03:14.609 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101302.log: /var/log/ceph/cluster2-client.mirror.105506.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105506.log.gz 2023-12-02T23:03:14.621 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120051.log 2023-12-02T23:03:14.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127627.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127627.log.gz 2023-12-02T23:03:14.625 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117889.log 2023-12-02T23:03:14.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.120051.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93061.log.gz 2023-12-02T23:03:14.628 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120051.log.gz 2023-12-02T23:03:14.630 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98877.log 2023-12-02T23:03:14.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117889.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117889.log.gz 2023-12-02T23:03:14.630 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107742.log 2023-12-02T23:03:14.633 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102075.log.gz 2023-12-02T23:03:14.634 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119761.log 2023-12-02T23:03:14.635 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107742.log: /var/log/ceph/cluster1-client.mirror.98877.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98877.log.gz 2023-12-02T23:03:14.635 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107742.log.gz 2023-12-02T23:03:14.636 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108490.log 2023-12-02T23:03:14.639 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119761.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119761.log.gz 2023-12-02T23:03:14.639 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100642.log 2023-12-02T23:03:14.639 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104659.log 2023-12-02T23:03:14.639 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100642.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100642.log.gz 2023-12-02T23:03:14.639 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121259.log 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104659.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.104659.log.gz 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108490.log.gz 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130190.log 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121259.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121259.log.gz 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117310.log 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130190.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130190.log.gz 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112551.log 2023-12-02T23:03:14.643 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93124.log 2023-12-02T23:03:14.646 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112551.log: /var/log/ceph/cluster2-client.mirror.117310.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117310.log.gz 2023-12-02T23:03:14.646 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100316.log 2023-12-02T23:03:14.646 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94438.log 2023-12-02T23:03:14.646 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100316.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100316.log.gz 2023-12-02T23:03:14.646 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101937.log 2023-12-02T23:03:14.647 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94438.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94438.log.gz 2023-12-02T23:03:14.647 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104981.log 2023-12-02T23:03:14.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101937.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101937.log.gz 2023-12-02T23:03:14.648 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118697.log 2023-12-02T23:03:14.651 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104981.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104981.log.gz 2023-12-02T23:03:14.652 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101486.log 2023-12-02T23:03:14.652 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112551.log.gz 2023-12-02T23:03:14.652 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126672.log 2023-12-02T23:03:14.654 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118697.log.gz 2023-12-02T23:03:14.654 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116560.log 2023-12-02T23:03:14.656 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126672.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126672.log.gz 2023-12-02T23:03:14.656 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.98635.log 2023-12-02T23:03:14.656 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101486.log: 80.4%/var/log/ceph/cluster2-client.mirror.116560.log: -- replaced with /var/log/ceph/cluster2-client.mirror.101486.log.gz 2023-12-02T23:03:14.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.98635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117165.log 2023-12-02T23:03:14.660 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116560.log.gz 2023-12-02T23:03:14.661 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127082.log 2023-12-02T23:03:14.663 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117165.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117165.log.gz 2023-12-02T23:03:14.664 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106380.log 2023-12-02T23:03:14.664 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.127082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127082.log.gz 2023-12-02T23:03:14.664 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98247.log 2023-12-02T23:03:14.665 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106380.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106380.log.gz 2023-12-02T23:03:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111262.log 2023-12-02T23:03:14.667 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98247.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98247.log.gz 2023-12-02T23:03:14.667 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107679.log 2023-12-02T23:03:14.668 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111262.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111262.log.gz 2023-12-02T23:03:14.669 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124990.log 2023-12-02T23:03:14.670 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107679.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.107679.log.gz 2023-12-02T23:03:14.670 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130064.log 2023-12-02T23:03:14.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124990.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124990.log.gz 2023-12-02T23:03:14.672 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104360.log 2023-12-02T23:03:14.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130064.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130064.log.gz 2023-12-02T23:03:14.672 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92620.log 2023-12-02T23:03:14.673 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104360.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.104360.log.gz 2023-12-02T23:03:14.674 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128971.log 2023-12-02T23:03:14.676 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92620.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93124.log.gz 2023-12-02T23:03:14.679 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108850.log 2023-12-02T23:03:14.680 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128971.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128971.log.gz 2023-12-02T23:03:14.680 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115259.log 2023-12-02T23:03:14.680 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108850.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108850.log.gz 2023-12-02T23:03:14.684 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129350.log 2023-12-02T23:03:14.686 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115259.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115259.log.gz 2023-12-02T23:03:14.686 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119513.log 2023-12-02T23:03:14.686 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129350.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129350.log.gz 2023-12-02T23:03:14.686 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115322.log 2023-12-02T23:03:14.687 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119513.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.92620.log.gz 2023-12-02T23:03:14.689 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119513.log.gz 2023-12-02T23:03:14.689 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118302.log 2023-12-02T23:03:14.691 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115322.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115322.log.gz 2023-12-02T23:03:14.691 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121214.log 2023-12-02T23:03:14.692 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118302.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118302.log.gz 2023-12-02T23:03:14.692 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120786.log 2023-12-02T23:03:14.695 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121214.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121214.log.gz 2023-12-02T23:03:14.695 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127753.log 2023-12-02T23:03:14.695 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120786.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120786.log.gz 2023-12-02T23:03:14.695 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107563.log 2023-12-02T23:03:14.695 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127753.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127753.log.gz 2023-12-02T23:03:14.696 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100375.log 2023-12-02T23:03:14.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107563.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107563.log.gz 2023-12-02T23:03:14.697 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95473.log 2023-12-02T23:03:14.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100375.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102497.log.gz 2023-12-02T23:03:14.706 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130904.log 2023-12-02T23:03:14.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95473.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.98635.log.gz 2023-12-02T23:03:14.713 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120921.log 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130904.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130904.log.gz 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120854.log 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120921.log: 86.7% 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95473.log.gz 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120921.log.gz 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97201.log 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120854.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120854.log.gz 2023-12-02T23:03:14.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89584.log 2023-12-02T23:03:14.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97201.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.97201.log.gz 2023-12-02T23:03:14.723 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123690.log 2023-12-02T23:03:14.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112506.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89584.log.gz 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121860.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123690.log.gz 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107583.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121860.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121860.log.gz 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112506.log.gz 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91272.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107583.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107583.log.gz 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111042.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.91272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114962.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102174.log 2023-12-02T23:03:14.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114962.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114962.log.gz 2023-12-02T23:03:14.725 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103140.log 2023-12-02T23:03:14.725 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111042.log.gz 2023-12-02T23:03:14.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102174.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102174.log.gz 2023-12-02T23:03:14.731 INFO:teuthology.orchestra.run.smithi170.stderr: 66.9% -- replaced with /var/log/ceph/cluster2-client.admin.91272.log.gz 2023-12-02T23:03:14.732 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98347.log 2023-12-02T23:03:14.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99721.log 2023-12-02T23:03:14.743 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98347.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98347.log.gz 2023-12-02T23:03:14.743 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109975.log 2023-12-02T23:03:14.743 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99721.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99721.log.gz 2023-12-02T23:03:14.748 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105980.log 2023-12-02T23:03:14.751 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109975.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109975.log.gz 2023-12-02T23:03:14.751 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102376.log 2023-12-02T23:03:14.755 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105980.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105980.log.gz 2023-12-02T23:03:14.755 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112972.log 2023-12-02T23:03:14.759 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102376.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102376.log.gz 2023-12-02T23:03:14.760 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130358.log 2023-12-02T23:03:14.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112972.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112972.log.gz 2023-12-02T23:03:14.761 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105823.log 2023-12-02T23:03:14.763 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96485.log.gz 2023-12-02T23:03:14.765 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130358.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130358.log.gz 2023-12-02T23:03:14.765 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115460.log 2023-12-02T23:03:14.765 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107075.log 2023-12-02T23:03:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115460.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115460.log.gz 2023-12-02T23:03:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101322.log 2023-12-02T23:03:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107075.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107075.log.gz 2023-12-02T23:03:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111648.log 2023-12-02T23:03:14.769 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105823.log.gz 2023-12-02T23:03:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101322.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.101322.log.gz 2023-12-02T23:03:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97109.log 2023-12-02T23:03:14.770 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118405.log 2023-12-02T23:03:14.771 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97109.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97109.log.gz 2023-12-02T23:03:14.771 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126338.log 2023-12-02T23:03:14.773 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118405.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111648.log.gz 2023-12-02T23:03:14.773 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103865.log 2023-12-02T23:03:14.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126338.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118405.log.gz 2023-12-02T23:03:14.774 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100155.log 2023-12-02T23:03:14.776 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103865.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103865.log.gz 2023-12-02T23:03:14.776 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94926.log 2023-12-02T23:03:14.777 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100155.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100155.log.gz 2023-12-02T23:03:14.777 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95042.log 2023-12-02T23:03:14.779 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94926.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126338.log.gz 2023-12-02T23:03:14.779 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121416.log 2023-12-02T23:03:14.779 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95042.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94926.log.gz 2023-12-02T23:03:14.779 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103239.log 2023-12-02T23:03:14.781 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121416.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121416.log.gz 2023-12-02T23:03:14.781 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94946.log 2023-12-02T23:03:14.781 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103239.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103239.log.gz 2023-12-02T23:03:14.782 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120516.log 2023-12-02T23:03:14.782 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94946.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94946.log.gz 2023-12-02T23:03:14.783 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126166.log 2023-12-02T23:03:14.783 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120516.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120516.log.gz 2023-12-02T23:03:14.784 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107012.log 2023-12-02T23:03:14.784 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126166.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126166.log.gz 2023-12-02T23:03:14.785 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122422.log 2023-12-02T23:03:14.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107012.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107012.log.gz 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.95042.log.gz 2023-12-02T23:03:14.787 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:14.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128299.log 2023-12-02T23:03:14.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126315.log 2023-12-02T23:03:14.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128299.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128299.log.gz 2023-12-02T23:03:14.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119679.log 2023-12-02T23:03:14.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126315.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126315.log.gz 2023-12-02T23:03:14.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110997.log 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119679.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122422.log.gz 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119679.log.gz 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108331.log 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102418.log 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110997.log.gz 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117269.log 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108331.log.gz 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97376.log 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117269.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117269.log.gz 2023-12-02T23:03:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103081.log 2023-12-02T23:03:14.804 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97376.log: /var/log/ceph/cluster2-client.mirror.102418.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97376.log.gz 2023-12-02T23:03:14.804 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126108.log 2023-12-02T23:03:14.804 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103081.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103081.log.gz 2023-12-02T23:03:14.804 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100135.log 2023-12-02T23:03:14.804 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126108.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126108.log.gz 2023-12-02T23:03:14.805 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96965.log 2023-12-02T23:03:14.818 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98387.log 2023-12-02T23:03:14.818 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96965.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.96965.log.gz 2023-12-02T23:03:14.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110973.log 2023-12-02T23:03:14.833 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98387.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98387.log.gz 2023-12-02T23:03:14.833 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121782.log 2023-12-02T23:03:14.840 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110973.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123973.log 2023-12-02T23:03:14.841 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110973.log.gz 2023-12-02T23:03:14.843 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121782.log: 86.9% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124032.log.gz 2023-12-02T23:03:14.843 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.121782.log.gz 2023-12-02T23:03:14.849 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104226.log 2023-12-02T23:03:14.854 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101302.log.gz 2023-12-02T23:03:14.855 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123973.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123973.log.gz 2023-12-02T23:03:14.862 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116229.log 2023-12-02T23:03:14.864 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104226.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104226.log.gz 2023-12-02T23:03:14.864 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90968.log 2023-12-02T23:03:14.865 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116229.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116229.log.gz 2023-12-02T23:03:14.866 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.90627.log 2023-12-02T23:03:14.867 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.90968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90968.log.gz 2023-12-02T23:03:14.867 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89797.log 2023-12-02T23:03:14.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.90627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126980.log 2023-12-02T23:03:14.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89797.log.gz 2023-12-02T23:03:14.872 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102233.log.gz 2023-12-02T23:03:14.880 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100933.log 2023-12-02T23:03:14.880 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126980.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114452.log 2023-12-02T23:03:14.880 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126980.log.gz 2023-12-02T23:03:14.880 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101116.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120831.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103355.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114452.log: /var/log/ceph/cluster1-client.mirror.120831.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120831.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126129.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103355.log: 87.4% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114452.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103355.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101116.log: /var/log/ceph/cluster2-client.mirror.100933.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100933.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101116.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 93.8% -5 --verbose -- /var/log/ceph/cluster2-client.admin.127795.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.102576.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116205.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126129.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98064.log 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127795.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127795.log.gz 2023-12-02T23:03:14.881 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102798.log 2023-12-02T23:03:14.882 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93146.log 2023-12-02T23:03:14.882 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116205.log.gz 2023-12-02T23:03:14.882 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107949.log 2023-12-02T23:03:14.882 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90547.log 2023-12-02T23:03:14.882 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107949.log: /var/log/ceph/cluster2-client.mirror.93146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107483.log 2023-12-02T23:03:14.894 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90547.log: /var/log/ceph/cluster2-client.mirror.102798.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107949.log.gz 2023-12-02T23:03:14.894 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125935.log 2023-12-02T23:03:14.894 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107483.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102798.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128467.log 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93146.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119472.log 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128467.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128467.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108645.log 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107483.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116872.log 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98064.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93104.log 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108645.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116872.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90547.log.gz 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96039.log 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126002.log 2023-12-02T23:03:14.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96039.log: /var/log/ceph/cluster2-client.mirror.119472.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119472.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr: 90.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108528.log 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116872.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109840.log 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108528.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126002.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108528.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103749.log 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109840.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109840.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128635.log 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103749.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103749.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106872.log 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128635.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128635.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108758.log 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96039.log.gz 2023-12-02T23:03:14.896 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99738.log 2023-12-02T23:03:14.910 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108758.log: /var/log/ceph/cluster2-client.mirror.106872.log: 89.7% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108758.log.gz 2023-12-02T23:03:14.910 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.106872.log.gz 2023-12-02T23:03:14.910 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102132.log 2023-12-02T23:03:14.910 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100807.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102132.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99738.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102132.log.gz 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107438.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121011.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106988.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107438.log: /var/log/ceph/cluster1-client.mirror.121011.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107438.log.gz 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121011.log.gz 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106988.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106988.log.gz 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101573.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101164.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101573.log: gzip -5 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101573.log.gz 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.128173.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100785.log 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101164.log.gz 2023-12-02T23:03:14.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128173.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128173.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118720.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100214.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100785.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98991.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100214.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118720.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112876.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98991.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98991.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111515.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112876.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112876.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95172.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111515.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111515.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99572.log 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95172.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93104.log.gz 2023-12-02T23:03:14.912 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95172.log.gz 2023-12-02T23:03:14.913 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91238.log 2023-12-02T23:03:14.913 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99572.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99572.log.gz 2023-12-02T23:03:14.915 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124924.log 2023-12-02T23:03:14.915 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.91238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95107.log 2023-12-02T23:03:14.915 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124924.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124924.log.gz 2023-12-02T23:03:14.915 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85689.log 2023-12-02T23:03:14.916 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95107.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.admin.91238.log.gz 2023-12-02T23:03:14.920 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95107.log.gz 2023-12-02T23:03:14.920 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97677.log 2023-12-02T23:03:14.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.85689.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85689.log.gz 2023-12-02T23:03:14.921 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105406.log 2023-12-02T23:03:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97677.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97677.log.gz 2023-12-02T23:03:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125278.log 2023-12-02T23:03:14.930 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105406.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105406.log.gz 2023-12-02T23:03:14.930 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93977.log 2023-12-02T23:03:14.931 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125278.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125278.log.gz 2023-12-02T23:03:14.931 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.95707.log 2023-12-02T23:03:14.932 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105269.log 2023-12-02T23:03:14.935 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.95707.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93977.log.gz 2023-12-02T23:03:14.949 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111122.log 2023-12-02T23:03:14.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105269.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105269.log.gz 2023-12-02T23:03:14.955 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111870.log 2023-12-02T23:03:14.969 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111122.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111122.log.gz 2023-12-02T23:03:14.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119699.log 2023-12-02T23:03:14.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111870.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111870.log.gz 2023-12-02T23:03:14.987 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89207.log 2023-12-02T23:03:15.002 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119699.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119699.log.gz 2023-12-02T23:03:15.006 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118831.log 2023-12-02T23:03:15.007 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89207.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89207.log.gz 2023-12-02T23:03:15.016 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96271.log 2023-12-02T23:03:15.017 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118831.log.gz 2023-12-02T23:03:15.020 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105751.log 2023-12-02T23:03:15.021 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96271.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96271.log.gz 2023-12-02T23:03:15.021 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94702.log 2023-12-02T23:03:15.024 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105751.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105751.log.gz 2023-12-02T23:03:15.030 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117331.log 2023-12-02T23:03:15.039 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94702.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94702.log.gz 2023-12-02T23:03:15.039 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118013.log 2023-12-02T23:03:15.043 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117331.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117331.log.gz 2023-12-02T23:03:15.043 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108802.log 2023-12-02T23:03:15.049 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118013.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118013.log.gz 2023-12-02T23:03:15.053 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110417.log 2023-12-02T23:03:15.053 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108802.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108802.log.gz 2023-12-02T23:03:15.054 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100484.log 2023-12-02T23:03:15.058 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110417.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110417.log.gz 2023-12-02T23:03:15.064 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118873.log 2023-12-02T23:03:15.064 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100484.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100484.log.gz 2023-12-02T23:03:15.064 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97247.log 2023-12-02T23:03:15.067 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118873.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118873.log.gz 2023-12-02T23:03:15.067 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93703.log 2023-12-02T23:03:15.067 INFO:teuthology.orchestra.run.smithi170.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.95707.log.gz 2023-12-02T23:03:15.073 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97247.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97247.log.gz 2023-12-02T23:03:15.074 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114496.log 2023-12-02T23:03:15.075 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113035.log 2023-12-02T23:03:15.075 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114496.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114496.log.gz 2023-12-02T23:03:15.077 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109586.log 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113035.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113035.log.gz 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.100844.log 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109586.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100135.log.gz 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94089.log 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109586.log.gz/var/log/ceph/cluster2-client.mirror.3.100844.log: 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120899.log 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112683.log 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120899.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120899.log.gz 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114299.log 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112683.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112683.log.gz 2023-12-02T23:03:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110867.log 2023-12-02T23:03:15.089 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114299.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114299.log.gz 2023-12-02T23:03:15.089 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100970.log 2023-12-02T23:03:15.089 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110867.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110867.log.gz 2023-12-02T23:03:15.089 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114741.log 2023-12-02T23:03:15.089 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100970.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100970.log.gz 2023-12-02T23:03:15.089 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97333.log 2023-12-02T23:03:15.091 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114741.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114741.log.gz 2023-12-02T23:03:15.091 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103944.log 2023-12-02T23:03:15.092 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.100844.log.gz 2023-12-02T23:03:15.093 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100622.log 2023-12-02T23:03:15.093 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103944.log: /var/log/ceph/cluster2-client.mirror.97333.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103944.log.gz 2023-12-02T23:03:15.094 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94109.log 2023-12-02T23:03:15.099 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97333.log.gz 2023-12-02T23:03:15.099 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124301.log 2023-12-02T23:03:15.100 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125814.log 2023-12-02T23:03:15.101 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124301.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124301.log.gz 2023-12-02T23:03:15.112 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117909.log 2023-12-02T23:03:15.112 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125814.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125814.log.gz 2023-12-02T23:03:15.117 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102554.log 2023-12-02T23:03:15.119 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117909.log: 93.4% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117909.log.gz 2023-12-02T23:03:15.121 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94089.log.gz 2023-12-02T23:03:15.122 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117517.log 2023-12-02T23:03:15.124 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102554.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102554.log.gz 2023-12-02T23:03:15.124 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119844.log 2023-12-02T23:03:15.124 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117517.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117517.log.gz 2023-12-02T23:03:15.124 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85556.log 2023-12-02T23:03:15.127 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119844.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119844.log.gz 2023-12-02T23:03:15.127 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94300.log 2023-12-02T23:03:15.127 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.85556.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85556.log.gz 2023-12-02T23:03:15.127 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103197.log 2023-12-02T23:03:15.130 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.94300.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.94300.log.gz 2023-12-02T23:03:15.130 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110330.log 2023-12-02T23:03:15.130 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103197.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103197.log.gz 2023-12-02T23:03:15.130 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114810.log 2023-12-02T23:03:15.130 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110330.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110330.log.gz 2023-12-02T23:03:15.131 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98748.log 2023-12-02T23:03:15.133 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114810.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114810.log.gz 2023-12-02T23:03:15.133 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103025.log 2023-12-02T23:03:15.133 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98748.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98748.log.gz 2023-12-02T23:03:15.134 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129602.log 2023-12-02T23:03:15.150 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103025.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103140.log.gz 2023-12-02T23:03:15.151 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92459.log 2023-12-02T23:03:15.151 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129602.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129602.log.gz 2023-12-02T23:03:15.151 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107033.log 2023-12-02T23:03:15.151 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101776.log 2023-12-02T23:03:15.152 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107033.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102418.log.gz 2023-12-02T23:03:15.152 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106510.log 2023-12-02T23:03:15.152 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101776.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107033.log.gz 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111745.log 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101776.log.gz/var/log/ceph/cluster1-client.mirror.106510.log: 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94109.log.gz 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.117207.log 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.111745.log: -- replaced with /var/log/ceph/cluster1-client.mirror.106510.log.gz 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101223.log 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111745.log.gz 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120314.log 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118178.log 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.101223.log: /var/log/ceph/cluster1-client.mirror.120314.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120314.log.gz 2023-12-02T23:03:15.157 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117207.log.gz 2023-12-02T23:03:15.158 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104485.log 2023-12-02T23:03:15.159 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118178.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118178.log.gz 2023-12-02T23:03:15.159 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128719.log 2023-12-02T23:03:15.162 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104485.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.104485.log.gz 2023-12-02T23:03:15.163 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113254.log 2023-12-02T23:03:15.163 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128719.log.gz 2023-12-02T23:03:15.163 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118116.log 2023-12-02T23:03:15.164 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113254.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113254.log.gz 2023-12-02T23:03:15.165 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117682.log 2023-12-02T23:03:15.166 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118116.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118116.log.gz 2023-12-02T23:03:15.166 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128005.log 2023-12-02T23:03:15.167 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117682.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117682.log.gz 2023-12-02T23:03:15.168 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116999.log 2023-12-02T23:03:15.168 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128005.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128005.log.gz 2023-12-02T23:03:15.168 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98596.log 2023-12-02T23:03:15.170 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116999.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116999.log.gz 2023-12-02T23:03:15.170 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.112086.log 2023-12-02T23:03:15.171 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98596.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98596.log.gz 2023-12-02T23:03:15.171 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106163.log 2023-12-02T23:03:15.185 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.2.112086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119265.log 2023-12-02T23:03:15.186 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106163.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106163.log.gz 2023-12-02T23:03:15.191 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99338.log 2023-12-02T23:03:15.194 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119265.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119265.log.gz 2023-12-02T23:03:15.200 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92459.log.gz 2023-12-02T23:03:15.207 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.112086.log.gz 2023-12-02T23:03:15.207 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100442.log 2023-12-02T23:03:15.208 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99338.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99338.log.gz 2023-12-02T23:03:15.208 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102761.log 2023-12-02T23:03:15.209 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100442.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100442.log.gz 2023-12-02T23:03:15.210 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112796.log 2023-12-02T23:03:15.216 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102761.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.90627.log.gz 2023-12-02T23:03:15.222 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121034.log 2023-12-02T23:03:15.224 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112796.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112796.log.gz 2023-12-02T23:03:15.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128383.log 2023-12-02T23:03:15.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121034.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121034.log.gz 2023-12-02T23:03:15.248 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101359.log 2023-12-02T23:03:15.249 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128383.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128383.log.gz 2023-12-02T23:03:15.262 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108597.log 2023-12-02T23:03:15.263 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101359.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101359.log.gz 2023-12-02T23:03:15.288 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118137.log 2023-12-02T23:03:15.288 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108597.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108597.log.gz 2023-12-02T23:03:15.297 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130148.log 2023-12-02T23:03:15.309 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107607.log 2023-12-02T23:03:15.309 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130148.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130148.log.gz 2023-12-02T23:03:15.309 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118137.log.gz 2023-12-02T23:03:15.317 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102112.log 2023-12-02T23:03:15.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.98787.log 2023-12-02T23:03:15.330 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102112.log: 89.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107607.log.gz 2023-12-02T23:03:15.335 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109003.log 2023-12-02T23:03:15.340 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.98787.log: -- replaced with /var/log/ceph/cluster2-client.mirror.102112.log.gz 2023-12-02T23:03:15.354 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113529.log 2023-12-02T23:03:15.355 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109003.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109003.log.gz 2023-12-02T23:03:15.363 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125855.log 2023-12-02T23:03:15.377 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113529.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113529.log.gz 2023-12-02T23:03:15.380 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116716.log 2023-12-02T23:03:15.383 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125855.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125855.log.gz 2023-12-02T23:03:15.384 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100375.log.gz 2023-12-02T23:03:15.385 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92900.log 2023-12-02T23:03:15.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116716.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116716.log.gz 2023-12-02T23:03:15.388 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.98787.log.gz 2023-12-02T23:03:15.389 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112993.log 2023-12-02T23:03:15.390 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103098.log 2023-12-02T23:03:15.399 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112993.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92900.log.gz 2023-12-02T23:03:15.400 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112993.log.gz 2023-12-02T23:03:15.400 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96377.log 2023-12-02T23:03:15.409 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103098.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103098.log.gz 2023-12-02T23:03:15.410 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.113576.log 2023-12-02T23:03:15.419 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96421.log 2023-12-02T23:03:15.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.2.113576.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93703.log.gz 2023-12-02T23:03:15.421 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96377.log.gz 2023-12-02T23:03:15.421 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109182.log 2023-12-02T23:03:15.431 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96421.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96421.log.gz 2023-12-02T23:03:15.431 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124945.log 2023-12-02T23:03:15.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109182.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124394.log 2023-12-02T23:03:15.433 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.109182.log.gz 2023-12-02T23:03:15.435 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124945.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124945.log.gz 2023-12-02T23:03:15.436 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128677.log 2023-12-02T23:03:15.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124394.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124394.log.gz 2023-12-02T23:03:15.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114275.log 2023-12-02T23:03:15.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128677.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128677.log.gz 2023-12-02T23:03:15.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-02T23:03:15.441 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114275.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114275.log.gz 2023-12-02T23:03:15.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106763.log 2023-12-02T23:03:15.442 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112372.log 2023-12-02T23:03:15.443 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106763.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106763.log.gz 2023-12-02T23:03:15.452 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125032.log 2023-12-02T23:03:15.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112372.log: 92.5% 89.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-02T23:03:15.463 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95356.log 2023-12-02T23:03:15.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125032.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.125032.log.gz 2023-12-02T23:03:15.469 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103662.log 2023-12-02T23:03:15.470 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95356.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112372.log.gz 2023-12-02T23:03:15.482 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95356.log.gz 2023-12-02T23:03:15.483 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.113576.log.gz 2023-12-02T23:03:15.485 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128887.log 2023-12-02T23:03:15.489 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103662.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103662.log.gz 2023-12-02T23:03:15.490 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119451.log 2023-12-02T23:03:15.490 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128887.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128887.log.gz 2023-12-02T23:03:15.490 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99425.log 2023-12-02T23:03:15.495 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119451.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100807.log.gz 2023-12-02T23:03:15.495 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91212.log 2023-12-02T23:03:15.500 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99425.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119451.log.gz 2023-12-02T23:03:15.500 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119885.log 2023-12-02T23:03:15.500 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.91212.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91212.log.gz 2023-12-02T23:03:15.500 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129055.log 2023-12-02T23:03:15.506 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119885.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102761.log.gz 2023-12-02T23:03:15.509 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120809.log 2023-12-02T23:03:15.509 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119885.log.gz 2023-12-02T23:03:15.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129055.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91391.log 2023-12-02T23:03:15.509 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.129055.log.gz 2023-12-02T23:03:15.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120809.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120809.log.gz 2023-12-02T23:03:15.510 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109047.log 2023-12-02T23:03:15.510 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91067.log 2023-12-02T23:03:15.512 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109047.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91391.log.gz 2023-12-02T23:03:15.515 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100214.log.gz 2023-12-02T23:03:15.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98167.log 2023-12-02T23:03:15.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106269.log 2023-12-02T23:03:15.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91067.log: /var/log/ceph/cluster2-client.mirror.98167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118566.log 2023-12-02T23:03:15.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106269.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98167.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109047.log.gz 2023-12-02T23:03:15.520 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106269.log.gz 2023-12-02T23:03:15.520 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:15.520 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100007.log 2023-12-02T23:03:15.521 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118566.log: /var/log/ceph/cluster2-client.mirror.100007.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100007.log.gz 2023-12-02T23:03:15.523 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118566.log.gz 2023-12-02T23:03:15.523 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100622.log.gz 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91067.log.gz 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91135.log 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114765.log 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119245.log 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91135.log.gz 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107145.log 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104119.log 2023-12-02T23:03:15.548 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119245.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114765.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107145.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119245.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107145.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92730.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104119.log: gzip -5 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104119.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.95955.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104024.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124151.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104024.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95955.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115436.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.124151.log: --verbose -- /var/log/ceph/cluster1-client.mirror.120201.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115436.log: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129896.log 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104024.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115436.log.gz 2023-12-02T23:03:15.549 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120201.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120201.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.101460.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129896.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124151.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129896.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99425.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93885.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117289.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130652.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.105083.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130652.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115870.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.130652.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.105083.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93885.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117289.log.gz 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106438.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108448.log 2023-12-02T23:03:15.550 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.106438.log: -- replaced with /var/log/ceph/cluster1-client.mirror.115870.log.gz 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94486.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106438.log.gz 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.90667.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95859.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.90667.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94486.log.gz 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122242.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.95859.log: -- /var/log/ceph/cluster2-client.mirror.97071.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108448.log.gz 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116075.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95859.log.gz 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122242.log: 87.1% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122242.log.gz 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.92730.log.gz/var/log/ceph/cluster2-client.mirror.97071.log: 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120584.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120876.log 2023-12-02T23:03:15.551 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116075.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97071.log.gz 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109292.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124536.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120584.log.gz 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120876.log.gz/var/log/ceph/cluster1-client.mirror.109292.log: 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109292.log.gz 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.admin.129686.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113013.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114054.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113013.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113013.log.gz 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89836.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129686.log: /var/log/ceph/cluster1-client.mirror.114054.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129686.log.gz 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114054.log.gz 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.101669.log 2023-12-02T23:03:15.552 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89836.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89836.log.gz 2023-12-02T23:03:15.553 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124347.log 2023-12-02T23:03:15.561 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.3.101669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112900.log 2023-12-02T23:03:15.561 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124347.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124347.log.gz 2023-12-02T23:03:15.563 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.105083.log.gz 2023-12-02T23:03:15.569 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91186.log 2023-12-02T23:03:15.570 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112900.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112900.log.gz 2023-12-02T23:03:15.573 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119596.log 2023-12-02T23:03:15.573 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97740.log 2023-12-02T23:03:15.575 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119596.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119596.log.gz 2023-12-02T23:03:15.576 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101181.log 2023-12-02T23:03:15.578 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97740.log: /var/log/ceph/cluster2-client.admin.91186.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91186.log.gz 2023-12-02T23:03:15.578 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98207.log 2023-12-02T23:03:15.579 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101181.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101181.log.gz 2023-12-02T23:03:15.579 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102438.log 2023-12-02T23:03:15.580 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98207.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98207.log.gz 2023-12-02T23:03:15.580 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127879.log 2023-12-02T23:03:15.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102438.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102438.log.gz 2023-12-02T23:03:15.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104611.log 2023-12-02T23:03:15.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127879.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127879.log.gz 2023-12-02T23:03:15.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113676.log 2023-12-02T23:03:15.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121394.log 2023-12-02T23:03:15.585 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113676.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113676.log.gz 2023-12-02T23:03:15.585 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94555.log 2023-12-02T23:03:15.585 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121394.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121394.log.gz 2023-12-02T23:03:15.588 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122130.log 2023-12-02T23:03:15.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94555.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.90667.log.gz 2023-12-02T23:03:15.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104611.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104611.log.gz 2023-12-02T23:03:15.588 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121950.log 2023-12-02T23:03:15.589 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122130.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97740.log.gz 2023-12-02T23:03:15.589 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122130.log.gz 2023-12-02T23:03:15.589 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122175.log 2023-12-02T23:03:15.589 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121950.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121950.log.gz 2023-12-02T23:03:15.591 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94555.log.gz 2023-12-02T23:03:15.591 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115826.log 2023-12-02T23:03:15.592 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122175.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122175.log.gz 2023-12-02T23:03:15.595 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102877.log 2023-12-02T23:03:15.595 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112818.log 2023-12-02T23:03:15.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102877.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115826.log.gz 2023-12-02T23:03:15.596 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97935.log 2023-12-02T23:03:15.601 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112818.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112818.log.gz 2023-12-02T23:03:15.601 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96139.log 2023-12-02T23:03:15.601 INFO:teuthology.orchestra.run.smithi170.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.101669.log.gz 2023-12-02T23:03:15.602 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112707.log 2023-12-02T23:03:15.603 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96139.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.96139.log.gz 2023-12-02T23:03:15.603 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116343.log 2023-12-02T23:03:15.605 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112707.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112707.log.gz 2023-12-02T23:03:15.605 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112049.log 2023-12-02T23:03:15.606 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116343.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102877.log.gz 2023-12-02T23:03:15.606 INFO:teuthology.orchestra.run.smithi170.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116343.log.gz 2023-12-02T23:03:15.607 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115051.log 2023-12-02T23:03:15.607 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118157.log 2023-12-02T23:03:15.608 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112049.log: /var/log/ceph/cluster1-client.mirror.115051.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112049.log.gz 2023-12-02T23:03:15.608 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125464.log 2023-12-02T23:03:15.610 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118157.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115051.log.gz 2023-12-02T23:03:15.614 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118157.log.gz 2023-12-02T23:03:15.614 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111832.log 2023-12-02T23:03:15.614 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108939.log 2023-12-02T23:03:15.614 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125464.log: /var/log/ceph/cluster2-client.mirror.111832.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111832.log.gz 2023-12-02T23:03:15.614 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125464.log.gz 2023-12-02T23:03:15.614 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104840.log 2023-12-02T23:03:15.616 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108939.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97935.log.gz 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108939.log.gz 2023-12-02T23:03:15.616 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:15.616 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122332.log 2023-12-02T23:03:15.617 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106076.log 2023-12-02T23:03:15.617 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104840.log: /var/log/ceph/cluster1-client.mirror.122332.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104840.log.gz 2023-12-02T23:03:15.617 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122332.log.gz 2023-12-02T23:03:15.617 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101381.log 2023-12-02T23:03:15.618 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100093.log 2023-12-02T23:03:15.618 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106076.log: /var/log/ceph/cluster2-client.mirror.101381.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106076.log.gz 2023-12-02T23:03:15.618 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102966.log 2023-12-02T23:03:15.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100093.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100093.log.gz 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.99425.log 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125501.log 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102966.log: /var/log/ceph/cluster1-client.mirror.2.99425.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102966.log.gz 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96223.log 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125501.log: 76.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125501.log.gz 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111214.log 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96223.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96223.log.gz 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121664.log 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111214.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111214.log.gz 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98447.log 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121664.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121664.log.gz 2023-12-02T23:03:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92920.log 2023-12-02T23:03:15.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98447.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98447.log.gz 2023-12-02T23:03:15.630 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117393.log 2023-12-02T23:03:15.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93502.log 2023-12-02T23:03:15.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117393.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117393.log.gz 2023-12-02T23:03:15.630 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90627.log 2023-12-02T23:03:15.632 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100295.log 2023-12-02T23:03:15.632 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106004.log 2023-12-02T23:03:15.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100295.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93502.log.gz 2023-12-02T23:03:15.635 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92920.log.gz 2023-12-02T23:03:15.641 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122631.log 2023-12-02T23:03:15.642 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106004.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90627.log.gz 2023-12-02T23:03:15.647 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101223.log.gz 2023-12-02T23:03:15.649 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113346.log 2023-12-02T23:03:15.649 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106004.log.gz 2023-12-02T23:03:15.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.122631.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122631.log.gz 2023-12-02T23:03:15.649 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118075.log 2023-12-02T23:03:15.651 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113346.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113346.log.gz 2023-12-02T23:03:15.651 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115412.log 2023-12-02T23:03:15.651 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118075.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118075.log.gz 2023-12-02T23:03:15.651 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102613.log 2023-12-02T23:03:15.653 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115412.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115412.log.gz 2023-12-02T23:03:15.653 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124901.log 2023-12-02T23:03:15.657 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.102613.log: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.102613.log.gz 2023-12-02T23:03:15.657 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100236.log 2023-12-02T23:03:15.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124901.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124901.log.gz 2023-12-02T23:03:15.659 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91450.log 2023-12-02T23:03:15.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100236.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100236.log.gz 2023-12-02T23:03:15.661 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99592.log 2023-12-02T23:03:15.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93821.log 2023-12-02T23:03:15.661 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99592.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99592.log.gz 2023-12-02T23:03:15.662 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120071.log 2023-12-02T23:03:15.663 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114544.log 2023-12-02T23:03:15.664 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.120071.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91450.log.gz 2023-12-02T23:03:15.664 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93821.log.gz 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120071.log.gz 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93615.log 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114544.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114544.log.gz 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107788.log 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125720.log 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107788.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93615.log.gz 2023-12-02T23:03:15.672 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107788.log.gz 2023-12-02T23:03:15.673 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91016.log 2023-12-02T23:03:15.673 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125720.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125720.log.gz 2023-12-02T23:03:15.673 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111084.log 2023-12-02T23:03:15.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.91016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108114.log 2023-12-02T23:03:15.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.111084.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111084.log.gz 2023-12-02T23:03:15.675 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130484.log 2023-12-02T23:03:15.675 INFO:teuthology.orchestra.run.smithi170.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.91016.log.gz 2023-12-02T23:03:15.686 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108114.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125935.log.gz 2023-12-02T23:03:15.689 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123457.log 2023-12-02T23:03:15.690 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130484.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130484.log.gz 2023-12-02T23:03:15.690 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118677.log 2023-12-02T23:03:15.690 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123457.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123457.log.gz 2023-12-02T23:03:15.691 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120336.log 2023-12-02T23:03:15.691 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118677.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118677.log.gz 2023-12-02T23:03:15.691 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94131.log 2023-12-02T23:03:15.692 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120336.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120336.log.gz 2023-12-02T23:03:15.692 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101954.log 2023-12-02T23:03:15.693 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.90587.log 2023-12-02T23:03:15.694 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101954.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94131.log.gz 2023-12-02T23:03:15.696 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101954.log.gz 2023-12-02T23:03:15.696 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121927.log 2023-12-02T23:03:15.696 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.3.90587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104888.log 2023-12-02T23:03:15.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121927.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121927.log.gz 2023-12-02T23:03:15.697 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117372.log 2023-12-02T23:03:15.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104888.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.104888.log.gz 2023-12-02T23:03:15.697 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99380.log 2023-12-02T23:03:15.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117372.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117372.log.gz 2023-12-02T23:03:15.699 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94153.log 2023-12-02T23:03:15.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85936.log 2023-12-02T23:03:15.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97290.log 2023-12-02T23:03:15.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.85936.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85936.log.gz 2023-12-02T23:03:15.733 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118761.log 2023-12-02T23:03:15.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97290.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97290.log.gz 2023-12-02T23:03:15.751 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103025.log.gz 2023-12-02T23:03:15.751 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114830.log 2023-12-02T23:03:15.754 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118761.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118761.log.gz 2023-12-02T23:03:15.754 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109095.log 2023-12-02T23:03:15.756 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114830.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114830.log.gz 2023-12-02T23:03:15.756 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90587.log 2023-12-02T23:03:15.756 INFO:teuthology.orchestra.run.smithi170.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108114.log.gz 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109095.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109095.log.gz 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108874.log 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101545.log 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108874.log: /var/log/ceph/cluster2-client.mirror.90587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113093.log 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108874.log.gz 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97353.log 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113093.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90587.log.gz 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113093.log.gz 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95580.log 2023-12-02T23:03:15.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97353.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.90587.log.gz 2023-12-02T23:03:15.766 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97353.log.gz 2023-12-02T23:03:15.770 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.99425.log.gz 2023-12-02T23:03:15.771 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95907.log 2023-12-02T23:03:15.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112639.log 2023-12-02T23:03:15.775 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95907.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95907.log.gz 2023-12-02T23:03:15.778 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101460.log.gz 2023-12-02T23:03:15.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110555.log 2023-12-02T23:03:15.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112639.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112639.log.gz 2023-12-02T23:03:15.800 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111721.log 2023-12-02T23:03:15.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110555.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110555.log.gz 2023-12-02T23:03:15.813 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105647.log 2023-12-02T23:03:15.814 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111721.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111721.log.gz 2023-12-02T23:03:15.823 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97720.log 2023-12-02T23:03:15.825 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105647.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105647.log.gz 2023-12-02T23:03:15.827 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120629.log 2023-12-02T23:03:15.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121349.log 2023-12-02T23:03:15.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120629.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120629.log.gz 2023-12-02T23:03:15.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100521.log 2023-12-02T23:03:15.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121349.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121349.log.gz 2023-12-02T23:03:15.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119575.log 2023-12-02T23:03:15.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100521.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100521.log.gz 2023-12-02T23:03:15.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94618.log 2023-12-02T23:03:15.833 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119575.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97720.log.gz 2023-12-02T23:03:15.834 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97267.log 2023-12-02T23:03:15.838 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94618.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119575.log.gz 2023-12-02T23:03:15.838 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99679.log 2023-12-02T23:03:15.844 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97267.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97267.log.gz 2023-12-02T23:03:15.844 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120471.log 2023-12-02T23:03:15.844 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99679.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99679.log.gz 2023-12-02T23:03:15.846 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-02T23:03:15.846 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120471.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120471.log.gz 2023-12-02T23:03:15.846 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94618.log.gz 2023-12-02T23:03:15.846 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128215.log 2023-12-02T23:03:15.846 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123825.log 2023-12-02T23:03:15.849 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-osd.2.log: /var/log/ceph/cluster2-client.admin.128215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128215.log.gz 2023-12-02T23:03:15.849 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106739.log 2023-12-02T23:03:15.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113165.log 2023-12-02T23:03:15.850 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106739.log: /var/log/ceph/cluster1-client.mirror.123825.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106739.log.gz 2023-12-02T23:03:15.850 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123825.log.gz 2023-12-02T23:03:15.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98929.log 2023-12-02T23:03:15.850 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113165.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113165.log.gz 2023-12-02T23:03:15.852 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105775.log 2023-12-02T23:03:15.852 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92642.log 2023-12-02T23:03:15.852 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98929.log.gz 2023-12-02T23:03:15.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105775.log: gzip -5/var/log/ceph/cluster2-client.mirror.92642.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.105775.log.gz 2023-12-02T23:03:15.853 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92642.log.gz 2023-12-02T23:03:15.854 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.126592.log 2023-12-02T23:03:15.856 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108549.log 2023-12-02T23:03:15.856 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103679.log 2023-12-02T23:03:15.856 INFO:teuthology.orchestra.run.smithi170.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126592.log.gz 2023-12-02T23:03:15.857 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108549.log: /var/log/ceph/cluster2-client.mirror.103679.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103679.log.gz 2023-12-02T23:03:15.857 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108549.log.gz 2023-12-02T23:03:15.858 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125893.log 2023-12-02T23:03:15.860 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116254.log 2023-12-02T23:03:15.860 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100600.log 2023-12-02T23:03:15.861 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116254.log: 89.6% 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125893.log.gz 2023-12-02T23:03:15.861 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116254.log.gz 2023-12-02T23:03:15.862 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99817.log 2023-12-02T23:03:15.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93791.log 2023-12-02T23:03:15.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99817.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99817.log.gz 2023-12-02T23:03:15.863 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100600.log.gz 2023-12-02T23:03:15.867 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.93791.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.93791.log.gz 2023-12-02T23:03:15.867 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96083.log 2023-12-02T23:03:15.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101818.log 2023-12-02T23:03:15.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.96083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122594.log 2023-12-02T23:03:15.872 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101818.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101381.log.gz 2023-12-02T23:03:15.874 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117599.log 2023-12-02T23:03:15.874 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.122594.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122594.log.gz 2023-12-02T23:03:15.874 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100696.log 2023-12-02T23:03:15.877 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117599.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117599.log.gz 2023-12-02T23:03:15.877 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114010.log 2023-12-02T23:03:15.891 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104007.log 2023-12-02T23:03:15.900 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114010.log: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124456.log 2023-12-02T23:03:15.904 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104007.log: -- replaced with /var/log/ceph/cluster2-client.mirror.114010.log.gz 2023-12-02T23:03:15.912 INFO:teuthology.orchestra.run.smithi170.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104007.log.gz 2023-12-02T23:03:15.919 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110949.log 2023-12-02T23:03:15.919 INFO:teuthology.orchestra.run.smithi170.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124456.log.gz 2023-12-02T23:03:15.928 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115031.log 2023-12-02T23:03:15.936 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110949.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110949.log.gz 2023-12-02T23:03:15.937 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103924.log 2023-12-02T23:03:15.943 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115031.log: 93.4% 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115031.log.gz 2023-12-02T23:03:15.943 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100696.log.gz 2023-12-02T23:03:15.943 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93929.log 2023-12-02T23:03:15.945 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103924.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103924.log.gz 2023-12-02T23:03:15.946 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97462.log 2023-12-02T23:03:15.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108961.log 2023-12-02T23:03:15.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97462.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93929.log.gz 2023-12-02T23:03:15.951 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97462.log.gz 2023-12-02T23:03:15.953 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129139.log 2023-12-02T23:03:15.955 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.108961.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108961.log.gz 2023-12-02T23:03:15.955 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95332.log 2023-12-02T23:03:15.955 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129139.log.gz 2023-12-02T23:03:15.956 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101996.log 2023-12-02T23:03:15.956 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95332.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95332.log.gz 2023-12-02T23:03:15.957 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99970.log 2023-12-02T23:03:15.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118447.log 2023-12-02T23:03:15.971 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99970.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99970.log.gz 2023-12-02T23:03:15.983 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95558.log 2023-12-02T23:03:15.984 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118447.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118447.log.gz 2023-12-02T23:03:15.991 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97029.log 2023-12-02T23:03:15.992 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95558.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95558.log.gz 2023-12-02T23:03:16.004 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105083.log 2023-12-02T23:03:16.014 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97029.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89711.log 2023-12-02T23:03:16.015 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105083.log: -- replaced with /var/log/ceph/cluster2-client.mirror.97029.log.gz 2023-12-02T23:03:16.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99780.log 2023-12-02T23:03:16.027 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89711.log.gz 2023-12-02T23:03:16.032 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100295.log.gz 2023-12-02T23:03:16.033 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121596.log 2023-12-02T23:03:16.048 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120741.log 2023-12-02T23:03:16.048 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121596.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121596.log.gz 2023-12-02T23:03:16.056 INFO:teuthology.orchestra.run.smithi170.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105083.log.gz 2023-12-02T23:03:16.057 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110688.log 2023-12-02T23:03:16.057 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120741.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120741.log.gz 2023-12-02T23:03:16.071 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100464.log 2023-12-02T23:03:16.072 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110688.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110688.log.gz 2023-12-02T23:03:16.083 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100501.log 2023-12-02T23:03:16.094 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103456.log 2023-12-02T23:03:16.094 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100501.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100501.log.gz 2023-12-02T23:03:16.102 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119327.log 2023-12-02T23:03:16.117 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117413.log 2023-12-02T23:03:16.118 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119327.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119327.log.gz 2023-12-02T23:03:16.132 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127006.log 2023-12-02T23:03:16.133 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117413.log: -- replaced with /var/log/ceph/cluster2-client.mirror.124536.log.gz 2023-12-02T23:03:16.134 INFO:teuthology.orchestra.run.smithi170.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117413.log.gz 2023-12-02T23:03:16.148 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.127006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127006.log.gz 2023-12-02T23:03:16.148 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92483.log 2023-12-02T23:03:16.161 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109912.log 2023-12-02T23:03:16.164 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92483.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.92483.log.gz 2023-12-02T23:03:16.170 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113466.log 2023-12-02T23:03:16.172 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109912.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109912.log.gz 2023-12-02T23:03:16.188 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105530.log 2023-12-02T23:03:16.189 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113466.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113466.log.gz 2023-12-02T23:03:16.213 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104820.log 2023-12-02T23:03:16.214 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105530.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105530.log.gz 2023-12-02T23:03:16.228 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104820.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125913.log 2023-12-02T23:03:16.230 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92111.log.gz 2023-12-02T23:03:16.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100580.log 2023-12-02T23:03:16.240 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125913.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104820.log.gz 2023-12-02T23:03:16.252 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101818.log.gz 2023-12-02T23:03:16.252 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114520.log 2023-12-02T23:03:16.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114472.log 2023-12-02T23:03:16.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114520.log: 89.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114520.log.gz 2023-12-02T23:03:16.253 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100580.log.gz 2023-12-02T23:03:16.253 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104912.log 2023-12-02T23:03:16.253 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.125913.log.gz 2023-12-02T23:03:16.253 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114472.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114472.log.gz 2023-12-02T23:03:16.255 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97654.log 2023-12-02T23:03:16.255 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104912.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.104912.log.gz 2023-12-02T23:03:16.255 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123399.log 2023-12-02T23:03:16.263 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97654.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97654.log.gz 2023-12-02T23:03:16.263 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123420.log 2023-12-02T23:03:16.263 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123399.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123399.log.gz 2023-12-02T23:03:16.263 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93393.log 2023-12-02T23:03:16.266 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123420.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123420.log.gz 2023-12-02T23:03:16.267 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125257.log 2023-12-02T23:03:16.267 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126802.log 2023-12-02T23:03:16.267 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125257.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125257.log.gz 2023-12-02T23:03:16.267 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97591.log 2023-12-02T23:03:16.268 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126802.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93393.log.gz 2023-12-02T23:03:16.268 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126802.log.gz 2023-12-02T23:03:16.268 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101138.log 2023-12-02T23:03:16.272 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97591.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99380.log.gz 2023-12-02T23:03:16.273 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101632.log 2023-12-02T23:03:16.273 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101138.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97591.log.gz 2023-12-02T23:03:16.274 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98725.log 2023-12-02T23:03:16.292 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105602.log 2023-12-02T23:03:16.293 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98725.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98725.log.gz 2023-12-02T23:03:16.306 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105731.log 2023-12-02T23:03:16.306 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.105602.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105602.log.gz 2023-12-02T23:03:16.308 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95580.log.gz 2023-12-02T23:03:16.318 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103256.log 2023-12-02T23:03:16.320 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105731.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105731.log.gz 2023-12-02T23:03:16.334 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107241.log 2023-12-02T23:03:16.334 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103256.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103256.log.gz 2023-12-02T23:03:16.342 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101545.log.gz 2023-12-02T23:03:16.346 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122377.log 2023-12-02T23:03:16.347 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107241.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107241.log.gz 2023-12-02T23:03:16.353 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94640.log 2023-12-02T23:03:16.356 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122377.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101996.log.gz 2023-12-02T23:03:16.361 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102053.log 2023-12-02T23:03:16.361 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94640.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122377.log.gz 2023-12-02T23:03:16.361 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94640.log.gz 2023-12-02T23:03:16.361 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103766.log 2023-12-02T23:03:16.361 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102053.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102053.log.gz 2023-12-02T23:03:16.361 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.90707.log 2023-12-02T23:03:16.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103766.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103766.log.gz 2023-12-02T23:03:16.368 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94534.log 2023-12-02T23:03:16.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.2.90707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129728.log 2023-12-02T23:03:16.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94534.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94534.log.gz 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110531.log 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129728.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129728.log.gz 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116278.log 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110531.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110531.log.gz 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111419.log 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116278.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116278.log.gz 2023-12-02T23:03:16.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122152.log 2023-12-02T23:03:16.372 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111419.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111419.log.gz 2023-12-02T23:03:16.372 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120159.log 2023-12-02T23:03:16.372 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122152.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122152.log.gz 2023-12-02T23:03:16.372 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123846.log 2023-12-02T23:03:16.372 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120159.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120159.log.gz 2023-12-02T23:03:16.372 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98187.log 2023-12-02T23:03:16.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123846.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123846.log.gz 2023-12-02T23:03:16.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128593.log 2023-12-02T23:03:16.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98187.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98187.log.gz 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89078.log 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128593.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128593.log.gz 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108427.log 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89078.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89078.log.gz 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89459.log 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108427.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108427.log.gz 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101838.log 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89459.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89459.log.gz 2023-12-02T23:03:16.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118811.log 2023-12-02T23:03:16.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101838.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101838.log.gz 2023-12-02T23:03:16.382 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108826.log 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118811.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118811.log.gz 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108283.log 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108826.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108826.log.gz 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95815.log 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108283.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108283.log.gz 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112755.log 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95815.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95815.log.gz 2023-12-02T23:03:16.387 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95707.log 2023-12-02T23:03:16.388 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112755.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112755.log.gz 2023-12-02T23:03:16.388 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108686.log 2023-12-02T23:03:16.392 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124010.log 2023-12-02T23:03:16.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108686.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108686.log.gz 2023-12-02T23:03:16.395 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118529.log 2023-12-02T23:03:16.396 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124010.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124010.log.gz 2023-12-02T23:03:16.396 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130778.log 2023-12-02T23:03:16.400 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118529.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118529.log.gz 2023-12-02T23:03:16.400 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102781.log 2023-12-02T23:03:16.400 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130778.log.gz 2023-12-02T23:03:16.401 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126023.log 2023-12-02T23:03:16.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102781.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102781.log.gz 2023-12-02T23:03:16.402 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123911.log 2023-12-02T23:03:16.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126023.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126023.log.gz 2023-12-02T23:03:16.403 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89335.log 2023-12-02T23:03:16.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123911.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123911.log.gz 2023-12-02T23:03:16.408 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115010.log 2023-12-02T23:03:16.408 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89335.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89335.log.gz 2023-12-02T23:03:16.408 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116831.log 2023-12-02T23:03:16.408 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115010.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115010.log.gz 2023-12-02T23:03:16.408 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.124188.log 2023-12-02T23:03:16.413 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116831.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.90707.log.gz 2023-12-02T23:03:16.415 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116831.log.gz 2023-12-02T23:03:16.415 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95404.log 2023-12-02T23:03:16.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.1.124188.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99780.log.gz 2023-12-02T23:03:16.417 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117703.log 2023-12-02T23:03:16.417 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95404.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95404.log.gz 2023-12-02T23:03:16.417 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90385.log 2023-12-02T23:03:16.417 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117703.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117703.log.gz 2023-12-02T23:03:16.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.90385.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90385.log.gz 2023-12-02T23:03:16.421 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113234.log 2023-12-02T23:03:16.429 INFO:teuthology.orchestra.run.smithi170.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.124188.log.gz 2023-12-02T23:03:16.430 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113697.log 2023-12-02T23:03:16.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125835.log 2023-12-02T23:03:16.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113697.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113697.log.gz 2023-12-02T23:03:16.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127201.log 2023-12-02T23:03:16.432 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113234.log.gz 2023-12-02T23:03:16.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120539.log 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1%/var/log/ceph/cluster1-client.admin.127201.log: -- replaced with /var/log/ceph/cluster2-client.mirror.125835.log.gz 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.127201.log.gz 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91050.log 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93591.log 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120539.log.gz 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.91050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119968.log 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126359.log 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119968.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119968.log.gz 2023-12-02T23:03:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110780.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126359.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126359.log.gz 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93591.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125340.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/cluster1-client.mirror.110780.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102455.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110780.log.gz 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125340.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125340.log.gz 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122040.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96109.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122040.log: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.91050.log.gz 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122040.log.gz 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102455.log.gz 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92579.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.96109.log: -- /var/log/ceph/cluster2-client.mirror.93660.log 2023-12-02T23:03:16.447 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.96109.log.gz/var/log/ceph/cluster1-client.mirror.92579.log: 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95644.log 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.93660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107525.log 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119307.log 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.95644.log.gz 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114673.log 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107525.log.gz 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119307.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119307.log.gz 2023-12-02T23:03:16.448 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106358.log 2023-12-02T23:03:16.453 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114673.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114673.log.gz 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107832.log 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106358.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.92579.log.gz 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109071.log 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107832.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106358.log.gz 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116491.log 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109071.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109071.log.gz 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.107832.log.gz 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129560.log 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116491.log.gz 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105689.log 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129560.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129560.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112924.log 2023-12-02T23:03:16.454 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.455 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112924.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112924.log.gz 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105689.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105689.log.gz 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115388.log 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112330.log 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115388.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115388.log.gz 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127669.log 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111238.log 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127669.log: /var/log/ceph/cluster2-client.mirror.112330.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127669.log.gz 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100464.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113117.log 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111238.log: gzip 87.4% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101096.log 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111238.log.gz 2023-12-02T23:03:16.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113117.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112330.log.gz 2023-12-02T23:03:16.462 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113117.log.gz 2023-12-02T23:03:16.462 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103160.log 2023-12-02T23:03:16.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101796.log 2023-12-02T23:03:16.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103160.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103160.log.gz 2023-12-02T23:03:16.462 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101096.log.gz 2023-12-02T23:03:16.462 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.102877.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101796.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101796.log.gz 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113721.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121529.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.102877.log: /var/log/ceph/cluster1-client.mirror.113721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127459.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100950.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112530.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111283.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98367.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121079.log 2023-12-02T23:03:16.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106338.log 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101523.log 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115119.log 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113721.log.gz 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.127459.log: /var/log/ceph/cluster2-client.mirror.119658.log 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127459.log.gz 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115119.log: /var/log/ceph/cluster2-client.mirror.100950.log: /var/log/ceph/cluster1-client.mirror.112530.log: /var/log/ceph/cluster2-client.mirror.111283.log: /var/log/ceph/cluster2-client.mirror.98367.log: /var/log/ceph/cluster1-client.mirror.121079.log: /var/log/ceph/cluster2-client.mirror.106338.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121079.log.gz 2023-12-02T23:03:16.469 INFO:teuthology.orchestra.run.smithi170.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98367.log.gz 2023-12-02T23:03:16.470 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106338.log.gz 2023-12-02T23:03:16.471 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101523.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101523.log.gz 2023-12-02T23:03:16.471 INFO:teuthology.orchestra.run.smithi170.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115119.log.gz 2023-12-02T23:03:16.471 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121529.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121529.log.gz 2023-12-02T23:03:16.472 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100950.log.gz 2023-12-02T23:03:16.473 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112530.log.gz 2023-12-02T23:03:16.473 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111283.log.gz 2023-12-02T23:03:16.473 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92213.log 2023-12-02T23:03:16.474 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93660.log.gz 2023-12-02T23:03:16.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119658.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119658.log.gz 2023-12-02T23:03:16.475 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91101.log 2023-12-02T23:03:16.476 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89251.log 2023-12-02T23:03:16.478 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110089.log 2023-12-02T23:03:16.478 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.92213.log.gz 2023-12-02T23:03:16.478 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.89251.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95707.log.gz 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91101.log.gz 2023-12-02T23:03:16.478 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.478 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89251.log.gz 2023-12-02T23:03:16.478 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123670.log 2023-12-02T23:03:16.479 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110089.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110089.log.gz 2023-12-02T23:03:16.479 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.126859.log 2023-12-02T23:03:16.481 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105960.log 2023-12-02T23:03:16.481 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.126859.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123670.log.gz 2023-12-02T23:03:16.481 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107390.log 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.105960.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.126859.log.gz 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105960.log.gz 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100192.log 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109140.log 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107390.log.gz 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108355.log 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109140.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100192.log.gz 2023-12-02T23:03:16.484 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101401.log 2023-12-02T23:03:16.487 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108355.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108355.log.gz 2023-12-02T23:03:16.487 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109140.log.gz 2023-12-02T23:03:16.487 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.124415.log 2023-12-02T23:03:16.487 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104591.log 2023-12-02T23:03:16.487 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101401.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101401.log.gz 2023-12-02T23:03:16.487 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96295.log 2023-12-02T23:03:16.489 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104591.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104591.log.gz 2023-12-02T23:03:16.489 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124663.log 2023-12-02T23:03:16.489 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96295.log: /var/log/ceph/cluster2-client.mirror.124415.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96295.log.gz 2023-12-02T23:03:16.490 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124415.log.gz 2023-12-02T23:03:16.490 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106317.log 2023-12-02T23:03:16.490 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.90475.log 2023-12-02T23:03:16.491 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124663.log: /var/log/ceph/cluster1-client.mirror.106317.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124663.log.gz 2023-12-02T23:03:16.491 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106317.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96165.log 2023-12-02T23:03:16.491 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.492 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114568.log 2023-12-02T23:03:16.494 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.90475.log: /var/log/ceph/cluster2-client.mirror.96165.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96165.log.gz 2023-12-02T23:03:16.494 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119369.log 2023-12-02T23:03:16.497 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114568.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114568.log.gz 2023-12-02T23:03:16.497 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129644.log 2023-12-02T23:03:16.497 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.119369.log: --verbose -- /var/log/ceph/cluster1-client.mirror.109316.log 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119369.log.gz 2023-12-02T23:03:16.497 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129644.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129644.log.gz 2023-12-02T23:03:16.497 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.497 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113576.log 2023-12-02T23:03:16.498 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109316.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109316.log.gz 2023-12-02T23:03:16.498 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100273.log 2023-12-02T23:03:16.502 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113576.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.102877.log.gz 2023-12-02T23:03:16.511 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94970.log 2023-12-02T23:03:16.512 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100273.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100273.log.gz 2023-12-02T23:03:16.514 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117455.log 2023-12-02T23:03:16.514 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94970.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94970.log.gz 2023-12-02T23:03:16.514 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120404.log 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117455.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117455.log.gz 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116302.log 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.113576.log.gz/var/log/ceph/cluster1-client.mirror.120404.log: 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120404.log.gz 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107304.log 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.92793.log 2023-12-02T23:03:16.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107304.log: /var/log/ceph/cluster1-client.mirror.116302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124870.log 2023-12-02T23:03:16.519 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116302.log.gz 2023-12-02T23:03:16.519 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93723.log 2023-12-02T23:03:16.519 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.107304.log.gz 2023-12-02T23:03:16.519 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103298.log 2023-12-02T23:03:16.519 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.125607.log 2023-12-02T23:03:16.520 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103298.log: /var/log/ceph/cluster1-client.mirror.1.92793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112659.log 2023-12-02T23:03:16.530 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.125607.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.125607.log.gz 2023-12-02T23:03:16.530 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110021.log 2023-12-02T23:03:16.530 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112659.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112659.log.gz 2023-12-02T23:03:16.530 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114368.log 2023-12-02T23:03:16.534 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110021.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110021.log.gz 2023-12-02T23:03:16.535 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129223.log 2023-12-02T23:03:16.543 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.114368.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114368.log.gz 2023-12-02T23:03:16.543 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99358.log 2023-12-02T23:03:16.545 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.129223.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.92793.log.gz 2023-12-02T23:03:16.545 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129223.log.gz 2023-12-02T23:03:16.545 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114078.log 2023-12-02T23:03:16.545 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99358.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99358.log.gz 2023-12-02T23:03:16.545 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97049.log 2023-12-02T23:03:16.547 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.114078.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114078.log.gz 2023-12-02T23:03:16.547 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119947.log 2023-12-02T23:03:16.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121124.log 2023-12-02T23:03:16.565 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119947.log: 90.3% 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97049.log.gz 2023-12-02T23:03:16.565 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98487.log 2023-12-02T23:03:16.565 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121124.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121124.log.gz 2023-12-02T23:03:16.566 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.102613.log 2023-12-02T23:03:16.569 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98487.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98487.log.gz 2023-12-02T23:03:16.570 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93639.log 2023-12-02T23:03:16.572 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.102613.log: -- replaced with /var/log/ceph/cluster2-client.mirror.119947.log.gz 2023-12-02T23:03:16.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107262.log 2023-12-02T23:03:16.588 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117186.log 2023-12-02T23:03:16.590 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107262.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107262.log.gz 2023-12-02T23:03:16.592 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93639.log.gz 2023-12-02T23:03:16.601 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102211.log 2023-12-02T23:03:16.602 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117186.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117186.log.gz 2023-12-02T23:03:16.618 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93461.log 2023-12-02T23:03:16.619 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102211.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102211.log.gz 2023-12-02T23:03:16.627 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95493.log 2023-12-02T23:03:16.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93461.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103456.log.gz 2023-12-02T23:03:16.631 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99987.log 2023-12-02T23:03:16.631 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95493.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93461.log.gz 2023-12-02T23:03:16.637 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95493.log.gz 2023-12-02T23:03:16.637 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103721.log 2023-12-02T23:03:16.638 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99987.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99987.log.gz 2023-12-02T23:03:16.638 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98550.log 2023-12-02T23:03:16.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97806.log 2023-12-02T23:03:16.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98550.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98550.log.gz 2023-12-02T23:03:16.657 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106892.log 2023-12-02T23:03:16.664 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97806.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101632.log.gz 2023-12-02T23:03:16.666 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117641.log 2023-12-02T23:03:16.666 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106892.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106892.log.gz 2023-12-02T23:03:16.666 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112482.log 2023-12-02T23:03:16.668 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117641.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117641.log.gz 2023-12-02T23:03:16.668 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122310.log 2023-12-02T23:03:16.670 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112482.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112482.log.gz 2023-12-02T23:03:16.671 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120494.log 2023-12-02T23:03:16.671 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97806.log.gz 2023-12-02T23:03:16.671 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.122310.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122310.log.gz 2023-12-02T23:03:16.671 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106245.log 2023-12-02T23:03:16.671 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120494.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120494.log.gz 2023-12-02T23:03:16.673 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124324.log 2023-12-02T23:03:16.673 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106245.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106245.log.gz 2023-12-02T23:03:16.673 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117930.log 2023-12-02T23:03:16.673 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124324.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124324.log.gz 2023-12-02T23:03:16.675 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106293.log 2023-12-02T23:03:16.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117930.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117930.log.gz 2023-12-02T23:03:16.676 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120246.log 2023-12-02T23:03:16.678 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106293.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106293.log.gz 2023-12-02T23:03:16.678 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103003.log 2023-12-02T23:03:16.678 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120246.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120246.log.gz 2023-12-02T23:03:16.678 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.96852.log 2023-12-02T23:03:16.680 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103003.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103003.log.gz 2023-12-02T23:03:16.680 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94598.log 2023-12-02T23:03:16.682 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.96852.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.96852.log.gz 2023-12-02T23:03:16.682 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109646.log 2023-12-02T23:03:16.688 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.94598.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94598.log.gz 2023-12-02T23:03:16.689 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109953.log 2023-12-02T23:03:16.692 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109646.log: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109646.log.gz 2023-12-02T23:03:16.692 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127057.log 2023-12-02T23:03:16.694 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109953.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109953.log.gz 2023-12-02T23:03:16.694 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116785.log 2023-12-02T23:03:16.695 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.127057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127057.log.gz 2023-12-02T23:03:16.695 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116764.log 2023-12-02T23:03:16.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116785.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116785.log.gz 2023-12-02T23:03:16.698 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109469.log 2023-12-02T23:03:16.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.116764.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116764.log.gz 2023-12-02T23:03:16.699 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110804.log 2023-12-02T23:03:16.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109469.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109469.log.gz 2023-12-02T23:03:16.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126737.log 2023-12-02T23:03:16.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110804.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110804.log.gz 2023-12-02T23:03:16.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119141.log 2023-12-02T23:03:16.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.126737.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126737.log.gz 2023-12-02T23:03:16.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116005.log 2023-12-02T23:03:16.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119141.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119141.log.gz 2023-12-02T23:03:16.707 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97153.log 2023-12-02T23:03:16.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.116005.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116005.log.gz 2023-12-02T23:03:16.709 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121506.log 2023-12-02T23:03:16.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97153.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97153.log.gz 2023-12-02T23:03:16.712 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98573.log 2023-12-02T23:03:16.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121506.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121506.log.gz 2023-12-02T23:03:16.712 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117951.log 2023-12-02T23:03:16.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98573.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98573.log.gz 2023-12-02T23:03:16.712 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119493.log 2023-12-02T23:03:16.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117951.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117951.log.gz 2023-12-02T23:03:16.714 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107720.log 2023-12-02T23:03:16.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119493.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119493.log.gz 2023-12-02T23:03:16.715 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111190.log 2023-12-02T23:03:16.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107720.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107720.log.gz 2023-12-02T23:03:16.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99013.log 2023-12-02T23:03:16.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111190.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111190.log.gz 2023-12-02T23:03:16.719 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99092.log 2023-12-02T23:03:16.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110137.log 2023-12-02T23:03:16.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99092.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99092.log.gz 2023-12-02T23:03:16.747 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117868.log 2023-12-02T23:03:16.747 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110137.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110137.log.gz 2023-12-02T23:03:16.761 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96853.log 2023-12-02T23:03:16.761 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117868.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117868.log.gz 2023-12-02T23:03:16.768 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100253.log 2023-12-02T23:03:16.768 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96853.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.96853.log.gz 2023-12-02T23:03:16.777 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111810.log 2023-12-02T23:03:16.777 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100253.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100253.log.gz 2023-12-02T23:03:16.796 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117620.log 2023-12-02T23:03:16.797 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111810.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111810.log.gz 2023-12-02T23:03:16.805 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120449.log 2023-12-02T23:03:16.805 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117620.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117620.log.gz 2023-12-02T23:03:16.817 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93723.log.gz 2023-12-02T23:03:16.819 INFO:teuthology.orchestra.run.smithi170.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.102613.log.gz 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94190.log 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120449.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120449.log.gz 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99075.log 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94190.log: 73.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94190.log.gz 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103335.log 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99075.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99075.log.gz 2023-12-02T23:03:16.823 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102270.log 2023-12-02T23:03:16.824 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103335.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103335.log.gz 2023-12-02T23:03:16.824 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110200.log 2023-12-02T23:03:16.826 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102270.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102270.log.gz 2023-12-02T23:03:16.826 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90992.log 2023-12-02T23:03:16.826 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110200.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110200.log.gz 2023-12-02T23:03:16.827 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112007.log 2023-12-02T23:03:16.828 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.90992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90992.log.gz 2023-12-02T23:03:16.828 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95274.log 2023-12-02T23:03:16.838 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.112007.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112007.log.gz 2023-12-02T23:03:16.839 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96356.log 2023-12-02T23:03:16.844 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95274.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96083.log.gz 2023-12-02T23:03:16.845 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118914.log 2023-12-02T23:03:16.845 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.96356.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.96356.log.gz 2023-12-02T23:03:16.845 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95274.log.gz 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115368.log 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118914.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118914.log.gz 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118590.log 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.115368.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115368.log.gz 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99837.log 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118590.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118590.log.gz 2023-12-02T23:03:16.850 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119100.log 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120113.log 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99837.log: /var/log/ceph/cluster2-client.mirror.119100.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99837.log.gz 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119803.log 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119100.log.gz 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121439.log 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119803.log: /var/log/ceph/cluster2-client.mirror.120113.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119803.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120113.log.gz 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80836.log 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121439.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121439.log.gz 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104635.log 2023-12-02T23:03:16.855 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.tmp-client.admin.80836.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80836.log.gz 2023-12-02T23:03:16.859 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101759.log 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92375.log 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101759.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.104635.log.gz 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.101759.log.gz 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126067.log 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111062.log 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92375.log.gz 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126067.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126067.log.gz 2023-12-02T23:03:16.860 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120966.log 2023-12-02T23:03:16.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111062.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111062.log.gz 2023-12-02T23:03:16.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111375.log 2023-12-02T23:03:16.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123932.log 2023-12-02T23:03:16.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120966.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120966.log.gz 2023-12-02T23:03:16.864 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124790.log 2023-12-02T23:03:16.865 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.123932.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123932.log.gz 2023-12-02T23:03:16.865 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113838.log 2023-12-02T23:03:16.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124790.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124790.log.gz 2023-12-02T23:03:16.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106691.log 2023-12-02T23:03:16.871 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113838.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113838.log.gz 2023-12-02T23:03:16.871 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124828.log 2023-12-02T23:03:16.872 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.106691.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106691.log.gz 2023-12-02T23:03:16.872 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95883.log 2023-12-02T23:03:16.873 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124828.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124828.log.gz 2023-12-02T23:03:16.873 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95248.log 2023-12-02T23:03:16.874 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95883.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95883.log.gz 2023-12-02T23:03:16.875 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126396.log 2023-12-02T23:03:16.876 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95248.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.95248.log.gz 2023-12-02T23:03:16.876 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107904.log 2023-12-02T23:03:16.877 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126396.log: 83.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126396.log.gz 2023-12-02T23:03:16.877 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128803.log 2023-12-02T23:03:16.879 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107904.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107904.log.gz 2023-12-02T23:03:16.879 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118955.log 2023-12-02T23:03:16.879 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128803.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128803.log.gz 2023-12-02T23:03:16.880 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125876.log 2023-12-02T23:03:16.883 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118955.log: /var/log/ceph/cluster2-client.mirror.111375.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118955.log.gz 2023-12-02T23:03:16.883 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103276.log 2023-12-02T23:03:16.885 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125876.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125876.log.gz 2023-12-02T23:03:16.885 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125743.log 2023-12-02T23:03:16.885 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103276.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103276.log.gz 2023-12-02T23:03:16.885 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109816.log 2023-12-02T23:03:16.887 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125743.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125743.log.gz 2023-12-02T23:03:16.887 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109645.log 2023-12-02T23:03:16.887 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.109816.log: 89.8% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109816.log.gz 2023-12-02T23:03:16.887 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91033.log 2023-12-02T23:03:16.888 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109645.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109645.log.gz 2023-12-02T23:03:16.889 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121905.log 2023-12-02T23:03:16.889 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.91033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90707.log 2023-12-02T23:03:16.890 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121905.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121905.log.gz 2023-12-02T23:03:16.891 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.111375.log.gz 2023-12-02T23:03:16.893 INFO:teuthology.orchestra.run.smithi170.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.91033.log.gz 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116365.log 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112237.log 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97634.log 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.112237.log: /var/log/ceph/cluster2-client.mirror.116365.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112237.log.gz 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110113.log 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97634.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116365.log.gz 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98787.log 2023-12-02T23:03:16.898 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110113.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110113.log.gz 2023-12-02T23:03:16.900 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118364.log 2023-12-02T23:03:16.900 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98787.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.98787.log.gz 2023-12-02T23:03:16.901 INFO:teuthology.orchestra.run.smithi170.stderr: 82.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90707.log.gz 2023-12-02T23:03:16.901 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118790.log 2023-12-02T23:03:16.902 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118364.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118364.log.gz 2023-12-02T23:03:16.902 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102739.log 2023-12-02T23:03:16.904 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.118790.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118790.log.gz 2023-12-02T23:03:16.904 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109792.log 2023-12-02T23:03:16.905 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102739.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102739.log.gz 2023-12-02T23:03:16.905 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85769.log 2023-12-02T23:03:16.907 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109792.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109792.log.gz 2023-12-02T23:03:16.907 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95516.log 2023-12-02T23:03:16.907 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.85769.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85769.log.gz 2023-12-02T23:03:16.907 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108898.log 2023-12-02T23:03:16.911 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95516.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97634.log.gz 2023-12-02T23:03:16.913 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95516.log.gz 2023-12-02T23:03:16.913 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125607.log 2023-12-02T23:03:16.915 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.108898.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108898.log.gz 2023-12-02T23:03:16.915 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127543.log 2023-12-02T23:03:16.915 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94862.log 2023-12-02T23:03:16.916 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.127543.log: /var/log/ceph/cluster1-client.mirror.125607.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127543.log.gz 75.9% 2023-12-02T23:03:16.916 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125607.log.gz 2023-12-02T23:03:16.916 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119348.log 2023-12-02T23:03:16.916 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92992.log 2023-12-02T23:03:16.917 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.94862.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94862.log.gz 2023-12-02T23:03:16.917 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92507.log 2023-12-02T23:03:16.917 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90217.log 2023-12-02T23:03:16.917 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92992.log.gz 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92507.log: /var/log/ceph/cluster2-client.mirror.119348.log: 89.3% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.92507.log.gz 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119348.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123890.log 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-osd.0.log: /var/log/ceph/cluster1-client.admin.90217.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90217.log.gz 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95150.log 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91084.log 2023-12-02T23:03:16.922 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123890.log.gz 2023-12-02T23:03:16.929 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95150.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95150.log.gz 2023-12-02T23:03:16.931 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120651.log 2023-12-02T23:03:16.935 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108307.log 2023-12-02T23:03:16.935 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91084.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120651.log.gz 2023-12-02T23:03:16.939 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/cluster1-client.mirror.108307.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108307.log.gz 2023-12-02T23:03:16.939 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.130946.log 2023-12-02T23:03:16.939 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106201.log 2023-12-02T23:03:16.939 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130946.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130946.log.gz 2023-12-02T23:03:16.940 INFO:teuthology.orchestra.run.smithi170.stderr: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91084.log.gz 2023-12-02T23:03:16.940 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127711.log 2023-12-02T23:03:16.940 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103177.log 2023-12-02T23:03:16.940 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.106201.log: /var/log/ceph/cluster2-client.admin.127711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.127711.log.gz 2023-12-02T23:03:16.942 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125052.log 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106201.log.gz 2023-12-02T23:03:16.942 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.942 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128425.log 2023-12-02T23:03:16.943 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.125052.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.125052.log.gz 2023-12-02T23:03:16.943 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115754.log 2023-12-02T23:03:16.945 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.128425.log: 89.5% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128425.log.gz 2023-12-02T23:03:16.945 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103177.log.gz 2023-12-02T23:03:16.945 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99280.log 2023-12-02T23:03:16.945 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117475.log 2023-12-02T23:03:16.949 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99280.log: /var/log/ceph/cluster1-client.mirror.115754.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103298.log.gz 2023-12-02T23:03:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121574.log 2023-12-02T23:03:16.951 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.117475.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101138.log.gz 2023-12-02T23:03:16.952 INFO:teuthology.orchestra.run.smithi170.stderr: 92.3% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117475.log.gz 2023-12-02T23:03:16.953 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.115754.log.gz 2023-12-02T23:03:16.953 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104140.log 2023-12-02T23:03:16.956 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121574.log: gzip -5 --verbose -- 87.0%/var/log/ceph/cluster2-client.mirror.104140.log: -- replaced with /var/log/ceph/cluster1-client.mirror.121574.log.gz 2023-12-02T23:03:16.956 INFO:teuthology.orchestra.run.smithi170.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104140.log.gz 2023-12-02T23:03:16.956 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.113769.log 2023-12-02T23:03:16.956 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103397.log 2023-12-02T23:03:16.956 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104044.log 2023-12-02T23:03:16.956 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113769.log.gz 2023-12-02T23:03:16.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.103397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110600.log 2023-12-02T23:03:16.957 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.104044.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103397.log.gz 2023-12-02T23:03:16.957 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93680.log 2023-12-02T23:03:16.957 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104044.log.gz 2023-12-02T23:03:16.958 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110600.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110600.log.gz 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.118566.log 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115187.log 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.93680.log: /var/log/ceph/cluster1-client.mirror.2.118566.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99321.log 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.115187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.94190.log 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115187.log.gz 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99321.log: gzip 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99321.log.gz 2023-12-02T23:03:16.960 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95536.log 2023-12-02T23:03:16.961 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.90547.log 2023-12-02T23:03:16.966 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95536.log: /var/log/ceph/cluster2-client.mirror.1.94190.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.94190.log.gz 2023-12-02T23:03:16.969 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95536.log.gz 2023-12-02T23:03:16.969 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106418.log 2023-12-02T23:03:16.970 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102359.log 2023-12-02T23:03:16.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.2.90547.log: /var/log/ceph/cluster2-client.mirror.106418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107125.log 2023-12-02T23:03:16.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.102359.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.102359.log.gz 2023-12-02T23:03:16.971 INFO:teuthology.orchestra.run.smithi170.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.118566.log.gz 2023-12-02T23:03:16.972 INFO:teuthology.orchestra.run.smithi170.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106418.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99614.log 2023-12-02T23:03:16.972 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.107125.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107125.log.gz 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93680.log.gz 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98327.log 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.99614.log: -- /var/log/ceph/cluster2-client.admin.85868.log 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.98327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113276.log 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.85868.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85868.log.gz 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98327.log.gz 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101855.log 2023-12-02T23:03:16.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.113276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98084.log 2023-12-02T23:03:16.978 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.101855.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101855.log.gz 2023-12-02T23:03:16.978 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99134.log 2023-12-02T23:03:16.980 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.98084.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113276.log.gz 2023-12-02T23:03:16.980 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119389.log 2023-12-02T23:03:16.985 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.99134.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98084.log.gz 2023-12-02T23:03:16.985 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.116116.log 2023-12-02T23:03:16.987 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119389.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119389.log.gz 2023-12-02T23:03:16.987 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95670.log 2023-12-02T23:03:16.988 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.116116.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.90547.log.gz 2023-12-02T23:03:17.001 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121551.log 2023-12-02T23:03:17.002 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.95670.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95670.log.gz 2023-12-02T23:03:17.014 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125443.log 2023-12-02T23:03:17.014 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121551.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121551.log.gz 2023-12-02T23:03:17.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130736.log 2023-12-02T23:03:17.027 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.125443.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125443.log.gz 2023-12-02T23:03:17.040 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95452.log 2023-12-02T23:03:17.040 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.130736.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130736.log.gz 2023-12-02T23:03:17.045 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.116116.log.gz 2023-12-02T23:03:17.045 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110756.log 2023-12-02T23:03:17.047 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.95452.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95452.log.gz 2023-12-02T23:03:17.047 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124811.log 2023-12-02T23:03:17.048 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.110756.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110756.log.gz 2023-12-02T23:03:17.048 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104203.log 2023-12-02T23:03:17.048 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.124811.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124811.log.gz 2023-12-02T23:03:17.050 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100333.log 2023-12-02T23:03:17.051 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.104203.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104203.log.gz 2023-12-02T23:03:17.051 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120764.log 2023-12-02T23:03:17.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.100333.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100333.log.gz 2023-12-02T23:03:17.054 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93193.log 2023-12-02T23:03:17.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.120764.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120764.log.gz 2023-12-02T23:03:17.054 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126629.log 2023-12-02T23:03:17.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.93193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113858.log 2023-12-02T23:03:17.055 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.126629.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126629.log.gz 2023-12-02T23:03:17.068 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110905.log 2023-12-02T23:03:17.068 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113858.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.93193.log.gz 2023-12-02T23:03:17.070 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113858.log.gz 2023-12-02T23:03:17.083 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111938.log 2023-12-02T23:03:17.085 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.110905.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110905.log.gz 2023-12-02T23:03:17.099 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121281.log 2023-12-02T23:03:17.153 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.111938.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111938.log.gz 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124870.log.gz 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109449.log 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.121281.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121281.log.gz 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123547.log 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.109449.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109449.log.gz 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113141.log 2023-12-02T23:03:17.156 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.123547.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123547.log.gz 2023-12-02T23:03:17.163 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124640.log 2023-12-02T23:03:17.163 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.113141.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113141.log.gz 2023-12-02T23:03:17.164 INFO:teuthology.orchestra.run.smithi170.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.90475.log.gz 2023-12-02T23:03:17.165 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97548.log 2023-12-02T23:03:17.167 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.124640.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124640.log.gz 2023-12-02T23:03:17.167 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103721.log.gz 2023-12-02T23:03:17.167 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107880.log 2023-12-02T23:03:17.172 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.97548.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97548.log.gz 2023-12-02T23:03:17.173 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93040.log 2023-12-02T23:03:17.173 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115894.log 2023-12-02T23:03:17.173 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.107880.log: /var/log/ceph/cluster1-client.mirror.93040.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107880.log.gz 2023-12-02T23:03:17.175 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97783.log 2023-12-02T23:03:17.175 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119555.log 2023-12-02T23:03:17.175 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.97783.log: /var/log/ceph/cluster1-client.mirror.115894.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115894.log.gz 2023-12-02T23:03:17.176 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.119555.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119555.log.gz 2023-12-02T23:03:17.181 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97783.log.gz 2023-12-02T23:03:17.184 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93040.log.gz 2023-12-02T23:03:17.223 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99280.log.gz 2023-12-02T23:03:17.321 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99614.log.gz 2023-12-02T23:03:17.388 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99013.log.gz 2023-12-02T23:03:17.394 INFO:teuthology.orchestra.run.smithi170.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-12-02T23:03:17.613 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94153.log.gz 2023-12-02T23:03:17.613 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99134.log.gz 2023-12-02T23:03:20.947 INFO:teuthology.orchestra.run.smithi012.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-02T23:03:21.057 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93168.log.gz 2023-12-02T23:03:21.721 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92664.log.gz 2023-12-02T23:03:21.788 INFO:teuthology.orchestra.run.smithi012.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-02T23:03:22.559 INFO:teuthology.orchestra.run.smithi012.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-02T23:03:22.561 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-02T23:03:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m9.975s 2023-12-02T23:03:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m28.194s 2023-12-02T23:03:22.562 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m0.987s 2023-12-02T23:04:41.230 INFO:teuthology.orchestra.run.smithi170.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-02T23:04:42.091 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-02T23:04:50.180 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-02T23:04:50.182 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-02T23:04:50.182 INFO:teuthology.orchestra.run.smithi170.stderr:real 1m37.569s 2023-12-02T23:04:50.182 INFO:teuthology.orchestra.run.smithi170.stderr:user 3m37.532s 2023-12-02T23:04:50.182 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m9.385s 2023-12-02T23:04:50.183 INFO:tasks.ceph:Archiving logs... 2023-12-02T23:04:50.183 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/remote/smithi012/log 2023-12-02T23:04:50.184 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-02T23:04:54.017 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/remote/smithi170/log 2023-12-02T23:04:54.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-02T23:05:26.258 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-02T23:05:26.271 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0c523c70ca7aa7b49aa5b461902b17d9e90c1f95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T23:05:26.272 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-02T23:05:26.272 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-02T23:05:26.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-02T23:05:26.611 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-02T23:05:26.625 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-02T23:05:26.626 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T23:05:26.628 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T23:05:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2023-12-02T23:05:26.645 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2023-12-02T23:05:26.648 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T23:05:26.648 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2023-12-02T23:05:26.648 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +2628us[+2630us] +/- 78ms 2023-12-02T23:05:26.648 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 +90us[ +92us] +/- 55ms 2023-12-02T23:05:26.648 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -2717us[-2717us] +/- 80ms 2023-12-02T23:05:26.648 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T23:05:26.695 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T23:05:26.695 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-12-02T23:05:26.695 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +2679us[+2713us] +/- 78ms 2023-12-02T23:05:26.695 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 +128us[ +163us] +/- 55ms 2023-12-02T23:05:26.695 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -2677us[-2643us] +/- 80ms 2023-12-02T23:05:26.695 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T23:05:26.696 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-02T23:05:26.708 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-02T23:05:26.709 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-02T23:05:26.723 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-02T23:05:26.833 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-02T23:05:27.015 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T23:05:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T23:05:27.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T23:05:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T23:05:27.089 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-02T23:05:27.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-02T23:05:27.122 INFO:teuthology.task.internal:Duration was 3513.711767 seconds 2023-12-02T23:05:27.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-02T23:05:27.136 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-02T23:05:27.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-02T23:05:27.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-02T23:05:27.191 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T23:05:27.219 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T23:05:27.644 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-02T23:05:27.644 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2023-12-02T23:05:27.645 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-02T23:05:27.670 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-12-02T23:05:27.670 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-02T23:05:27.690 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-02T23:05:27.690 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-02T23:05:27.712 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-02T23:05:27.759 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-02T23:05:27.760 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T23:05:27.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T23:05:27.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-02T23:05:27.935 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-02T23:05:27.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-02T23:05:27.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-02T23:05:27.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-02T23:05:28.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-02T23:05:28.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-02T23:05:28.038 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2023-12-02T23:05:28.071 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-12-02T23:05:28.087 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-02T23:05:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T23:05:28.123 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-02T23:05:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T23:05:28.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-02T23:05:28.155 INFO:teuthology.task.internal:Transferring archived files... 2023-12-02T23:05:28.156 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/remote/smithi012 2023-12-02T23:05:28.156 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-02T23:05:28.203 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805/remote/smithi170 2023-12-02T23:05:28.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-02T23:05:28.240 INFO:teuthology.task.internal:Removing archive directory... 2023-12-02T23:05:28.241 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-02T23:05:28.243 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-02T23:05:28.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-02T23:05:28.307 INFO:teuthology.task.internal:Not uploading archives. 2023-12-02T23:05:28.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-02T23:05:28.327 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-02T23:05:28.327 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-02T23:05:28.330 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-02T23:05:28.342 INFO:teuthology.orchestra.run.smithi012.stdout: 265782 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 23:05 /home/ubuntu/cephtest 2023-12-02T23:05:28.344 INFO:teuthology.orchestra.run.smithi170.stdout: 265505 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 23:05 /home/ubuntu/cephtest 2023-12-02T23:05:28.345 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-02T23:05:28.362 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-02T23:05:28.377 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi170 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T23:05:28.377 INFO:teuthology.nuke:Checking targets against current locks 2023-12-02T23:05:28.405 DEBUG:teuthology.nuke:shortname: smithi012 2023-12-02T23:05:28.405 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T23:05:28.429 DEBUG:teuthology.nuke:shortname: smithi170 2023-12-02T23:05:28.429 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T23:05:28.433 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 21:55:17.244701', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXvhR6JCsTwPdNXSIOZvx45StfGsMUDwgQgZDSRvgDFGOjSjKQY32w4gnwhnr3iQ8P2nxx3y/yZOma8AInV3tM='} 2023-12-02T23:05:28.459 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-02_21:54:05-rbd-wip-dis-testing-distro-default-smithi/7475805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 21:55:17.246069', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAO9CBiefD2/lvxCVlSKINHzBVGThnPqas2AV49ObW4POGhsYMBoxblDlHCPGauDDzzxu0VRdL2bZtXX0ZVQWQ0='} 2023-12-02T23:05:28.485 INFO:teuthology.orchestra.console.smithi012:Power off 2023-12-02T23:05:28.486 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-02T23:05:28.509 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2023-12-02T23:05:28.510 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:28.528 INFO:teuthology.orchestra.console.smithi170:Power off 2023-12-02T23:05:28.528 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-02T23:05:28.538 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-12-02T23:05:28.551 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-12-02T23:05:28.551 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:28.572 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-02T23:05:32.540 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:32.573 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:32.649 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-12-02T23:05:32.682 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-02T23:05:36.653 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:36.683 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:36.763 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-12-02T23:05:36.793 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-12-02T23:05:36.893 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-12-02T23:05:40.765 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T23:05:40.873 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2023-12-02T23:05:40.974 INFO:teuthology.orchestra.console.smithi012:Power off completed 2023-12-02T23:05:41.078 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 3513.711767435074 failure_reason: 'Command failed on smithi170 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json''' flavor: default owner: scheduled_dis@teuthology status: fail success: false 2023-12-02T23:05:41.079 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-02T23:05:41.175 INFO:teuthology.run:FAIL